Ground State Sign-Changing Solution for Schrödinger-Poisson System with Critical Growth

Abstract

This article is devoted to study the nonlinear Schrödinger-Poisson system with pure power nonlinearities

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+u+ \phi u=|u|^{p-1}u+|u|^4u, &{}x\in {\mathbb {R}}^3, \\ -\Delta \phi = u^2, &{}x\in {\mathbb {R}}^3, \end{array} \right. \end{aligned}$$

where $4< p<5$. By employing constraint variational method and a variant of the classical deformation lemma, we show the existence of one ground state sign-changing solution with precisely two nodal domains, which improves and generalizes the existing results by Wang, Zhang and Guan (J. Math. Anal. Appl. 479 (2019), 2284–2301).

1 Introduction

In present paper, we deal with the Schrödinger-Poisson system with critical growth

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+ u+ \phi u=|u|^{p-1}u+|u|^4u, &{}x\in {\mathbb {R}}^3, \\ -\Delta \phi =u^2, &{}x\in {\mathbb {R}}^3, \end{array} \right. \end{aligned}$$

(1.1)

where $4< p<5$. It is a special form of the more general system as follows

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+V(x)u+K(x)\phi u=f(x,u), &{}x\in {\mathbb {R}}^3, \\ -\Delta \phi =K(x)u^2, &{}x\in {\mathbb {R}}^3, \end{array} \right. \end{aligned}$$

(1.2)

where $V, K\in C({\mathbb {R}}^3,{\mathbb {R}})$, $f\in C({\mathbb {R}}^3\times {\mathbb {R}},{\mathbb {R}})$. As quoted by Boenci and Vartunato in [6], system (1.2) works as a solitary wave model for describing the interaction between a nonlinear fixed Schrödinger equation and an electrostatic field. Another different justification of system (1.2) can be found also in [26], where it appears in semiconductor theory to model the evolution of an electron ensemble in a semiconductor crystal. For further details of the physical background of system (1.2), we refer the reader to the papers [2, 3] and the references therein.

As far as system (1.2) is concerned, due to the appearance of the term $K(x)\phi u$, it is viewed as being nonlocal and is no longer a pointwise identity. This observation brings mathematical challenges to the analyses, and at the same time makes the study of such a problem particularly interesting. Under different conditions of potential functions V(x) and K(x), many authors have already obtained the existence and nonexistence of positive solutions, multiple solutions, ground state solutions, radial and non-radical solutions and semiclassical states to system (1.2), see e.g. [2,3,4, 10, 12, 17, 24, 25, 34, 35] and the references listed therein.

In present paper, we are interested in the existence of sign-changing solutions of system (1.1), which is a very interesting subject and has gained many attentions more recently. In fact, several abstract theories and methods have been established for the existence of sign-changing solutions to system (1.2), for example by employing a dynamical approach together with a limit procedure (Ianni [16]), constructing invariant sets and descending flow (Liu et al. [21]), applying variational methods together with the Brouwer degree theory (Wang and Zhou [30]), combining constraint variational method and quantitative deformation lemma (Shuai and Wang [27], Chen and Tang [11], Wang et al. [29]), choosing appropriate minimizing sequence of sign-changing solutions constraint on bounded domain [1, 5], verifying the (PS)-condition (Zhong and Tang [36]). For more discussions on the existence of sign-changing solutions of system (1.2), we refer the reader to [15, 19, 33] and the references mentioned therein.

Since the embedding $H^1({\mathbb {R}}^3)\hookrightarrow L^6({\mathbb {R}}^3)$ is not compact, the investigations on Schrödinger-Poisson system with critical growth are more complicated and interesting from the mathematical point view. As far as we know, there are few results on sign-changing solutions for the Schröginger-Poisson system with critical growth, see [15, 29, 33, 36]. In these works, Huang et al. [15] considered the Schrödinger-Poisson system

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+u+K(x)\phi u=a(x)|u|^4u+\mu h(x)u, &{}x\in {\mathbb {R}}^3, \\ -\Delta \phi =K(x)u^2, &{}x\in {\mathbb {R}}^3, \end{array} \right. \end{aligned}$$

(1.3)

where $\mu$ is a positive constant, K(x), a(x) and h(x) are nonnegative functions in ${\mathbb {R}}^3$. Under suitable assumptions on potentials, they proved that system (1.3) has a pair of sign-changing solutions in $H^1({\mathbb {R}}^3)\times {\mathcal {D}}^{1,2}({\mathbb {R}}^3)$. In addition, Zhang [33] also focused on the Schrödinger-Poisson system with critical growth

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+u+K(x)\phi u=a(x)|u|^{p-1}u+u^5, &{}x\in {\mathbb {R}}^3, \\ -\Delta \phi =K(x)u^2, &{}x\in {\mathbb {R}}^3, \end{array} \right. \end{aligned}$$

(1.4)

with $3<p<5$ and the potentials satisfying some decay rate assumptions, he obtained the existence of ground state and sign-changing solutions of system (1.4). However, as pointed out in [36], actually these two works mentioned above only studied the case that system (1.3) and system (1.4) are not involved the nonlocal terms, that is, $K(x)\equiv 0$. In light of this discovery, Zhong and Tang [36] studied the following system

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+u+K(x)\phi u=\lambda h(x)u+|u|^4u, &{}x\in {\mathbb {R}}^3, \\ -\Delta \phi =K(x)u^2, &{}x\in {\mathbb {R}}^3, \end{array} \right. \end{aligned}$$

(1.5)

where $0<\lambda <\lambda _1$, $\lambda _1$ is the first eigenvalue of the problem $-\Delta u+u=\lambda h(x)u$ in $H^1({\mathbb {R}}^3)$ and the weight functions K(x), h(x) satisfy the following conditions:

$(H_K)$:

$K\in L^2({\mathbb {R}}^3)\cap L^{\infty }({\mathbb {R}}^3)\setminus \{0\}$ for some $p\in [2,+\infty )$ and K(x) is nonnegative;

$(H_{h_1})$:

$h\in L^{\frac{3}{2}}({\mathbb {R}}^3)\setminus \{0\}$ is nonnegative;

$(H_{h_2})$:

there exist $\rho >0$ and $\alpha >0$ such that $h(x)\ge C|x|^{-\alpha }$ for $|x|<\rho$.

By using the constraint variational method and quantitative deformation lemma, they showed that system (1.5) possesses at least one ground state sign-changing solution for each $0<\lambda <\lambda _1$ and its energy is strictly larger than twice that of ground state solution. As far as we know, the latest result about the sign-changing solutions of system (1.2) is obtained in [29]. Explicitly, Wang et al. considered the following system

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+V(x)u+\lambda \phi u=|u|^4u+\mu f(u), &{}x\in {\mathbb {R}}^3, \\ -\Delta \phi =u^2, &{}x\in {\mathbb {R}}^3, \end{array} \right. \end{aligned}$$

(1.6)

and, by restricting the parameter $\mu >0$ large enough, obtained the existence of ground state sign-changing solution for the case that f is of subcritical. Moreover, the authors also studied the asymptotic behavior of the sign-changing solutions of system (1.6) as the parameter $\lambda \rightarrow 0$. Here, it must be pointed out that the parameter $\mu >0$ large enough plays a vital role for their argument.

Consequently, a natural question is that if system (1.6) possesses sign-changing solutions without any restriction on the parameter $\mu >0$. In present paper, we give one affirmative answer to this question partially. Actually, we focus our attention on system (1.1) with $4<p<5$, and show the existence of ground state sign-changing solution. Before stating the main result, we introduce some necessary notations. Denote by $H^1({\mathbb {R}}^3)$ the usual Sobolev space with the inner product and norm

$$\begin{aligned} \langle u,v\rangle =\int _{{\mathbb {R}}^3}(\nabla u\cdot \nabla v+uv)dx,\quad \Vert u\Vert =\Bigl (\int _{{\mathbb {R}}^3}(|\nabla u|^2+u^2)dx\Bigr )^{\frac{1}{2}}, \end{aligned}$$

and

$$\begin{aligned} {\mathcal {D}}^{1,2}({\mathbb {R}}^3)=\{u\in L^6({\mathbb {R}}^3):\nabla u\in L^2({\mathbb {R}}^3)\} \end{aligned}$$

with the norm

$$\begin{aligned} \Vert u\Vert _{{\mathcal {D}}^{1,2}({\mathbb {R}}^3)}=\Bigl (\int _{{\mathbb {R}}^3}|\nabla u|^2dx\Bigr )^{\frac{1}{2}}. \end{aligned}$$

$\Vert \cdot \Vert _s$ ($1\le s \le \infty$) is the norm of usual Lebesgue space $L^s({\mathbb {R}}^3)$ and S is the best Sobolev constant for the embedding of ${\mathcal {D}}^{1,2}\hookrightarrow L^6({\mathbb {R}}^3)$, i.e.,

$$\begin{aligned} S:=\inf _{u\in {\mathcal {D}}^{1,2}({\mathbb {R}}^3)\setminus \{0\}}\frac{\int _{{\mathbb {R}}^3}|\nabla u|^2dx}{\Vert u\Vert _6^2}. \end{aligned}$$

(1.7)

Due to the fact that our system (1.1) is autonomous, it is usual to discuss the existence of solutions in radial space $H^1_r({\mathbb {R}}^3)$, that is,

$$\begin{aligned} H_r^1({\mathbb {R}}^3)=\{u\in H^1({\mathbb {R}}^3):u(x)=u(|x|)\}. \end{aligned}$$

Since the embedding $H^1({\mathbb {R}}^3)\hookrightarrow L^p({\mathbb {R}}^3)$ ($2\le p\le 6$) is continuous, then the embedding $H^1_r({\mathbb {R}}^3)\hookrightarrow L^p({\mathbb {R}}^3)$ is also continuous, that is, there exist $C_p>0$ such that

$$\begin{aligned} \Vert u\Vert _p\le C_p\Vert u\Vert ,\quad \forall u\in H^1_r({\mathbb {R}}^3),\,\, p\in [2,6]. \end{aligned}$$

(1.8)

Moreover, the embedding $H^1_r({\mathbb {R}}^3)\hookrightarrow L^p({\mathbb {R}}^3)$ is compact for $p\in (2,6)$, see [31].

It is obvious that the technique used in [29] cannot be adopted any more. Fortunately, with the help of the methods in [36], we can successfully overcome the difficulties caused by the absence of parameter. Since our work is based on variational methods, it is necessary to transform system (1.1) into a single Schrödinger equation with the nonlocal term. Recalling the Lax-Milgram theorem, for each $u\in H^1_r({\mathbb {R}}^3)$, there exists a unique $\phi _u\in {\mathcal {D}}^{1,2}({\mathbb {R}}^3)$ such that $-\triangle \phi _u=u^2$. Inserting this $\phi _u$ into the first equation of system (1.1), we obtain

$$\begin{aligned} -\Delta u+u+\phi _u u=|u|^{p-1}u+|u|^4u,\ \ \ u\in H^1_r({\mathbb {R}}^3), \end{aligned}$$

(1.9)

where

$$\begin{aligned} \phi _u(x)=\frac{1}{4\pi }\int _{{\mathbb {R}}^3}\frac{u^2(y)}{|x-y|}dy. \end{aligned}$$

(1.10)

Define the corresponding energy functional $I:H^1_r({\mathbb {R}}^3)\rightarrow {\mathbb {R}}$ to system (1.1) as follows:

$$\begin{aligned} \begin{aligned} I(u)&=\frac{1}{2}\int _{{\mathbb {R}}^{3}}(|\nabla u|^2+u^2)dx+\frac{1}{4}\int _{{\mathbb {R}}^3}\phi _{u}u^2dx-\frac{1}{p+1}\int _{{\mathbb {R}}^3}|u|^{p+1}dx \\&\ \ \ -\frac{1}{6}\int _{{\mathbb {R}}^3}|u|^6dx,\ \ \forall u\in H^1_r({\mathbb {R}}^3). \end{aligned} \end{aligned}$$

(1.11)

Then, $I\in C^{1}(H^1_r({\mathbb {R}}^3),{\mathbb {R}})$ and its Gâteaux derivative is given by

$$\begin{aligned} \begin{aligned} \langle I'(u),\varphi \rangle&=\int _{{\mathbb {R}}^3}(\nabla u\cdot \nabla \varphi +u\varphi )dx+\int _{{\mathbb {R}}^3}\phi _{u}u\varphi dx-\int _{{\mathbb {R}}^3}|u|^{p-1}u\varphi dx\\&\ \ \ -\int _{{\mathbb {R}}^3}|u|^4u\varphi dx,\ \ \forall \varphi \in H^1_r({\mathbb {R}}^3). \end{aligned} \end{aligned}$$

(1.12)

For notational convenience, we shall denote

$$\begin{aligned} L_{\phi _u}(u)=\int _{{\mathbb {R}}^3}\phi _{u}u^2dx=\frac{1}{4\pi }\int _{{\mathbb {R}}^3}\int _{{\mathbb {R}}^3}\frac{u^2(x)u^2(y)}{|x-y|}dxdy. \end{aligned}$$

(1.13)

As is well known, weak solutions for (1.9) correspond to critical points of the functional I. Here, we call that u is a weak solution of (1.9) if $u\in H^1_r({\mathbb {R}}^3)$ satisfies $\langle I'(u),\varphi \rangle =0$ for any $\varphi \in H^1_r({\mathbb {R}}^3)$. Moreover, if u is a solution of (1.9) with $u^\pm \not \equiv 0$, then u is called a sign-changing solution (nodal solution) of (1.9), where

$$\begin{aligned} u^+(x)=\max \{u(x),0\}\quad \text{ and } \quad u^-(x)=\min \{u(x),0\}. \end{aligned}$$

A solution is called a ground state solution if its energy is minimal among all nontrivial solutions.

In order to obtain ground state sign-changing solutions of system (1.1), inspired by [36], we show that the energy functional I satisfies the (PS)-condition at the minimization level m constrained on the following nodal set:

$$\begin{aligned} {\mathcal {M}}:=\{u\in H^1_r({\mathbb {R}}^3):u^\pm \ne 0,\ \ \langle I'(u),u^+\rangle =0=\langle I'(u),u^-\rangle \}, \end{aligned}$$

namely,

$$\begin{aligned} m:=\inf \{I(u):u\in {\mathcal {M}}\}, \end{aligned}$$

(1.14)

see Lemma 3.2 below. To estimate the energy of ground state sign-changing solution, we define the following Nehari manifold associated with system (1.1)

$$\begin{aligned} {\mathcal {N}}:=\{u\in H^1_r({\mathbb {R}}^3) \setminus \{0\}:\langle I'(u),u\rangle =0\}, \end{aligned}$$

and let

$$\begin{aligned} c:=\inf \{I(u):u\in {\mathcal {N}}\}. \end{aligned}$$

(1.15)

Now we are in the position to state our main result.

Theorem 1.1

System (1.1) with $4<p<5$ possesses one radially ground state sign-changing solution with precisely two nodal domains such that $m>2c$.

Remark 1.2

Theorem 1.1 implies that we have answered the natural question raised above. However, our approach is still not universal, due to the reason that it could not be applied for the case that that $3<p\le 4$ in system (1.1). Indeed, throughout the paper, except Lemma 3.3, the other preliminary results are valid for $3<p<5$. Explicitly, $4<p<5$ is only used to obtain the inequality (3.30), which is impossible for $3<p\le 4$. Consequently, it is worth exploring new techniques to discus sign-changing solutions for $3<p\le 4$ in system (1.1), even for more general case $1<p\le 3$.

In what follows, we discuss some difficulties need to be solved for our problem. First of all, as usual, since we are dealing with the problem in the whole space ${\mathbb {R}}^3$, the lack of compactness needs to be overcome. Secondly, from (1.13) and Fubini’s theorem, we see that

$$\begin{aligned} L_{\phi _{u^-}}(u^+)=\int _{{\mathbb {R}}^3}\phi _{u^-}(u^+)^2dx=\int _{{\mathbb {R}}^3}\phi _{u^+}(u^-)^2dx=L_{\phi _{u^+}}(u^-). \end{aligned}$$

Then, it deduces the following decompositions

$$\begin{aligned} I(u)= & {} I(u^+)+I(u^-)+\frac{1}{2}L_{\phi _{u^+}}(u^-), \end{aligned}$$

(1.16)

$$\begin{aligned} \langle I'(u),u^+\rangle= & {} \langle I'(u^+),u^+\rangle +L_{\phi _{u^-}}(u^+), \end{aligned}$$

(1.17)

and

$$\begin{aligned} \langle I'(u),u^-\rangle =\langle I'(u^-),u^-\rangle +L_{\phi _{u^+}}(u^-). \end{aligned}$$

(1.18)

According to (1.17) and (1.18), one can easily observe that if u is a sign-changing solution of system (1.1), then both the functions $u^\pm$ do not belong to the Nehari manifold. Therefore, the usual methods used to prove the existence of sign-changing solutions for semilinear local problems can not be used here. In addition, since we add $|u|^{p-1} u$ as a perturbation for the critical growth, the techniques adopted in [36] can not be applied directly to system (1.1). These difficulties make the problem more complex.

To overcome the difficulties mentioned above, in this article, we choose $H_r^1\left( {\mathbb {R}}^3\right)$ as the energy space because the embedding $H_r^1\left( {\mathbb {R}}^3\right) \hookrightarrow L^s\left( {\mathbb {R}}^3\right) (2<s<6)$ is compact, and study the energy functional I on a neighbourhood ${\mathbf {U}}$ (see (3.1)) of the nodal set ${\mathcal {M}}$. Then, after some subtle estimates for the energy functional I on ${\mathbf {U}}$, we successfully check the (PS)-condition about the minimization level m, see Lemmas 3.1–3.3 below.

Remark 1.3

Because our system is autonomous, the compactness of $H^1_r({\mathbb {R}}^3)\hookrightarrow L^s({\mathbb {R}}^3)$ ($2<s<6$) plays a vital role in checking the convergence of bounded (PS) sequences, see Lemma 3.2 below. However, one could not achieve this point in $H^1({\mathbb {R}}^3)$. In fact, up to now, we have not seen the literature dealing with system (1.2) in non-radial framework for the autonomous situation. Therefore, as pointed out in [13], see its Remark 1.1, for autonomous Schrödinger-Poisson system it is still an open problem to prove the existence of sign-changing solutions in non-radial setting.

The remainder of this paper is structured as follows. In the next Sect. 2, we present some preliminary results to pave the way for obtaining the ground state sign-changing solution. In Sect. 3, we are devoted to finish the proof of Theorem 1.1.

Throughout the paper, we use C to denote universal positive constants.

2 Preliminaries Results

In this section, we show the following lemmas which will play crucial roles in the sequel. We first list some properties that the function $\phi _u$ satisfies, see [24, Lemma 2.1].

Lemma 2.1

For the function $\phi _u$ defined in (1.10), one has

• (i) $\phi _u\ge 0$, $\forall u\in H^1({\mathbb {R}}^3)$;

• (ii) $\phi _{tu}=t^2\phi _u$, $\forall t>0$ and $u\in H^1({\mathbb {R}}^3)$;

• (iii) if $u_n \rightharpoonup u$ in $H^1_r({\mathbb {R}}^3)$, then, $\phi _{u_n} \rightarrow \phi _u$ in $D^{1,2}({\mathbb {R}}^3)$ and

  $$\begin{aligned} \int _{{\mathbb {R}}^3}\phi _{u_n}u^2_n dx\rightarrow \int _{{\mathbb {R}}^3}\phi _u u^2 dx; \end{aligned}$$

• (iv) there is a constant C such that

  $$\begin{aligned} \int _{{\mathbb {R}}^3}\phi _u u^2 dx\le \Vert \phi _u\Vert _6\Vert u\Vert _{\frac{12}{5}}^2\le C \Vert u\Vert ^4. \end{aligned}$$

To seek a minimizer of the energy functional I on ${\mathcal {M}}$, the following lemma illustrates that the set ${\mathcal {M}}$ is nonempty.

Lemma 2.2

For any $u\in H^1_r({\mathbb {R}}^3)$ with $u^\pm \ne 0$, there exists a unique pair $(s_u,t_u)$, with $s_u,t_u>0$ such that $s_u u^++t_u u^-\in {\mathcal {M}}$. Furthermore, we have the following relationship $I(s_u u^++t_u u^-)=\max _{s,t\ge 0}I(su^++tu^-)$. In addition, if $\langle I'(u), u^\pm \rangle \le 0$, then $s_u, t_u\le 1$.

Proof

Based on our nonlinearity, it directly follows from the procedure of Lemma 2.1 in [29]. So we omit the details. $\square$

Denote by $(s_u,t_u)$ the unique pair of positive numbers obtained from Lemma 2.2. We have proved that $s_u$ and $t_u$ are well defined. Moreover, we can also get the following properties.

Lemma 2.3

For any $u\in H^1_r({\mathbb {R}}^3)$ with $u^\pm \ne 0$, there hold

• (i) the functionals s, t are continuous in $H^1_r({\mathbb {R}}^3)$;

• (ii) $s_{u_n}\rightarrow \infty$ if $u_n^+\rightarrow 0$ in $H^1_r({\mathbb {R}}^3)$ as $n\rightarrow \infty$; $t_{u_n}\rightarrow \infty$ if $u_n^-\rightarrow 0$ in $H^1_r({\mathbb {R}}^3)$ as $n\rightarrow \infty$;

• (iii) if $\{u_n\}\subset {\mathcal {M}}$, $\lim _{n\rightarrow \infty }I(u_n)=m$, then $m>0$ and $\Lambda _1\le \Vert u_n^\pm \Vert \le \Lambda _2$ for some $\Lambda _1$, $\Lambda _2>0$.

Proof

(i) Take a sequence $\{u_n\}\in H^1_r({\mathbb {R}}^3)$ such that $u_n\rightarrow u$ in $H^1_r({\mathbb {R}}^3)$, then we have $u_n^\pm \rightarrow u^\pm$ in $H^1_r({\mathbb {R}}^3)$. By Lemma 2.2, there exist $(s_{u_n},t_{u_n})$ and $(s_u,t_u)$ such that $s_{u_n}u_n^++t_{u_n}u_n^-\in {\mathcal {M}}$ and $s_uu^++t_uu^-\in {\mathcal {M}}$. By the definition of ${\mathcal {M}}$, it yields that

$$\begin{aligned} \left\{ \begin{array}{ll} \Vert u_n^+\Vert ^2+s^2_{u_n} L_{\phi _{u_n^+}}(u_n^+)+t^2_{u_n} L_{\phi _{u_n^-}}(u_n^+)=s^{p-1}_{u_n}\displaystyle \int _{{\mathbb {R}}^3}|u_n^+|^{p+1}dx +s_{u_n}^4\displaystyle \int _{{\mathbb {R}}^3}|u_n^+|^6dx,\\ \Vert u_n^-\Vert ^2+t^2_{u_n} L_{\phi _{u_n^-}}(u_n^-)+s^2_{u_n} L_{\phi _{u_n^+}}(u_n^-)=t^{p-1}_{u_n}\displaystyle \int _{{\mathbb {R}}^3}|u_n^-|^{p+1}dx +t_{u_n}^4\displaystyle \int _{{\mathbb {R}}^3}|u_n^-|^6dx. \end{array} \right. \end{aligned}$$

(2.1)

We claim that $\{s_{u_n}\}$ and $\{t_{u_n}\}$ are bounded in ${\mathbb {R}}^+$. In fact, by contradiction, there holds $s_{u_n}\rightarrow \infty$ as $n\rightarrow \infty$. Then, it follows from $u_n^\pm \rightarrow u^\pm \ne 0$ in $H^1_r({\mathbb {R}}^3)$ and the first equality of (2.1) that $\frac{t_{u_n}}{s_{u_n}}\rightarrow \infty$ as $n\rightarrow \infty$, which is in contradiction with the second equality of (2.1). Therefore, passing if necessary to a subsequence, still denoted by $\{s_{u_n}\}$ and $\{t_{u_n}\}$, we can assume that there exists a pair nonnegative number $(s_0,t_0)$ such that

$$\begin{aligned} \lim _{n\rightarrow \infty }s_{u_n}=s_0 \quad \text{ and }\quad \lim _{n\rightarrow \infty }t_{u_n}=t_0. \end{aligned}$$

Taking the limit $n\rightarrow \infty$ in (2.1), we see that

$$\begin{aligned} \left\{ \begin{array}{ll} \Vert u^+\Vert ^2+s^2_0 L_{\phi _{u^+}}(u^+)+t^2_0 L_{\phi _{u^-}}(u^+)=s^{p-1}_0\displaystyle \int _{{\mathbb {R}}^3}|u^+|^{p+1}dx +s_0^4\displaystyle \int _{{\mathbb {R}}^3}|u^+|^6dx,\\ \Vert u^-\Vert ^2+t^2_0 L_{\phi _{u^-}}(u^-)+s^2_0 L_{\phi _{u^+}}(u^-)=t^{p-1}_0\displaystyle \int _{{\mathbb {R}}^3}|u^-|^{p+1}dx +t_0^4\displaystyle \int _{{\mathbb {R}}^3}|u^-|^6dx. \end{array} \right. \end{aligned}$$

Since $u^\pm \ne 0$, we have $s_0,t_0>0$, which means that $s_0u^++t_0u^-\in {\mathcal {M}}$. From the uniqueness of $(s_u,t_u)$, we derive that $s_u=s_0$ and $t_u=t_0$. This completes the proof of (i).

(ii) We just need to prove $s_{u_n}\rightarrow \infty$ if $u_n^+\rightarrow 0$ in $H^1_r({\mathbb {R}}^3)$ as $n\rightarrow \infty$, and the other one follows from the same argument. Suppose that, there exists $M>0$ such that $s_{u_n}\le M$. Using (1.8), one has

$$\begin{aligned} s_{u_n}^4\int _{{\mathbb {R}}^3}|u_n^+|^6dx\le C\Vert u_n^+\Vert ^6=o(\Vert u_n^+\Vert ^2) \end{aligned}$$

and

$$\begin{aligned} s_{u_n}^{p-1}\int _{{\mathbb {R}}^3}|u_n^+|^{p+1}dx\le C\Vert u_n^+\Vert ^{p+1}=o(\Vert u_n^+\Vert ^2). \end{aligned}$$

Therefore, it gives that

$$\begin{aligned} \begin{aligned} \frac{\langle I'(s_{u_n}u_n^+ +t_{u_n}u_n^-), s_{u_n}u_n^+\rangle }{s_{u_n}^2}&\ge \Vert u_n^+\Vert ^2 +s_{u_n}^2 L_{\phi _{u_n^+}}(u_n^+)+t_{u_n}^2 L_{\phi _{u_n^-}}(u_n^+)-o(\Vert u_n^+\Vert ^2)\\&\ge \Vert u_n^+\Vert ^2-o(\Vert u_n^+\Vert ^2)>0. \end{aligned} \end{aligned}$$

Hence, we are lead to a contradiction since $s_{u_n}u_n^++t_{u_n}u_n^-\in {\mathcal {M}}$.

(iii) On the one hand, since $\{u_n\}\subset {\mathcal {M}}$, we have

$$\begin{aligned} \Vert u_n^\pm \Vert ^2+L_{\phi _{u_n}}(u_n^\pm )=\int _{{\mathbb {R}}^3}|u_n^\pm |^{p+1}dx+\int _{{\mathbb {R}}^3}|u_n^\pm |^6dx. \end{aligned}$$

Then, from (1.8), we obtain

$$\begin{aligned} \begin{aligned} \Vert u_n^\pm \Vert ^2&\le \int _{{\mathbb {R}}^3}|u_n^\pm |^{p+1}dx+\int _{{\mathbb {R}}^3}|u_n^\pm |^6dx\\&\le C(\Vert u_n^\pm \Vert ^{p+1}+ \Vert u_n^\pm \Vert ^6), \end{aligned} \end{aligned}$$

which means that there exists $\Lambda _1>0$ such that $\Vert u_n^\pm \Vert \ge \Lambda _1>0$. On the other hand, from $\{u_n\}\subset {\mathcal {M}}\subset {\mathcal {N}}$, one has

$$\begin{aligned} \begin{aligned} m+o(1)&=I(u_n)=I(u_n)-\frac{1}{p+1}\langle I'(u_n),u_n\rangle \\&=\Bigl (\frac{1}{2}-\frac{1}{p+1}\Bigr )\Vert u_n\Vert ^2+\Bigl (\frac{1}{4}-\frac{1}{p+1}\Bigr )L_{\phi _{u_n}}(u_n) -\Bigl (\frac{1}{6}-\frac{1}{p+1}\Bigr )\int _{{\mathbb {R}}^3}|u_n|^6dx\\&\ge \Bigl (\frac{1}{2}-\frac{1}{p+1}\Bigr )\Vert u_n\Vert ^2, \end{aligned} \end{aligned}$$

which signifies that $m>0$ and $\{u_n\}$ is bounded in $H^1_r({\mathbb {R}}^3)$. That is to say there exists $\Lambda _2>0$ such that $\Lambda _1\le \Vert u_n^\pm \Vert \le \Lambda _2$. This concludes the proof. $\square$

With the exception of the previous conclusions, to establish the existence of ground state sign-changing solution, we also need the following lemma which can be derived from [20, Lemma 3.1] and [32, Theorem 1.2].

Lemma 2.4

(i) For any $u\in H^1_r({\mathbb {R}}^3)\setminus \{0\}$, there exists a unique ${\widetilde{s}}_u>0$ such that ${\widetilde{s}}_uu\in {\mathcal {N}}$. Moreover,

$$\begin{aligned} I({\widetilde{s}}_uu)=\max _{s\ge 0}I(su). \end{aligned}$$

(ii) System (1.1) has a positive ground state solution $u_0\in {\mathcal {N}}$ such that $I(u_0)=c$ and $c\in \Bigl (0,\frac{1}{3}S^{\frac{3}{2}}\Bigr )$.

Remark 2.5

Since system (1.1) is equivalent to Eq.(1.9), applying for [18, Theorem 1.11] directly, we have $u_0 \in L^{\infty }({\mathbb {R}}^3)$ and $C^{1,\alpha }_{loc}({\mathbb {R}}^3)$ for some $0<\alpha <1$. The boundedness and regularity of $u_0$ are also very important for the proof of Theorem 1.1 in Sect. 3.

3 Sign-Changing Solution

In this section, we examine the existence of ground state sign-changing solution for problem (1.1). Before starting the proof, inspired by [9], we recall the following notations. Let P denote the cone of nonnegative functions in $H^1_r({\mathbb {R}}^3)$, $Q=[0,1]\times [0,1]$ and $\Sigma$ be the set of continuous maps $\sigma$ such that for each $s,t\in [0,1]$,

• (a) $\sigma (s,0)=0,\sigma (0,t)\in P$ and $\sigma (1,t)\in -P$;

• (b) $(I\circ \sigma )(s,1)\le 0$, $\frac{\int _{{\mathbb {R}}^3}|\sigma (s,1)|^{p+1}dx+\int _{{\mathbb {R}}^3}|\sigma (s,1)|^6dx}{\Vert \sigma (s,1)\Vert ^2+ L_{\phi _{\sigma (s,1)}}(\sigma (s,1))}\ge 2$.

For any $u\in H^1_r({\mathbb {R}}^3)$ with $u^\pm \ne 0$, we can take $\sigma (s,t)=\gamma t(1-s)u^++\gamma tsu^-\in \Sigma$ for $\gamma$ large enough and this implies that $\Sigma \ne \emptyset$. Define the functional

$$\begin{aligned} \xi (u,v)= {\left\{ \begin{array}{ll} \frac{\int _{{\mathbb {R}}^3}|u|^{p+1}dx+\int _{{\mathbb {R}}^3}|u|^6dx}{\Vert u\Vert ^2+L_{\phi _{u}}(u)+ L_{\phi _{v}}(u)}, &{}u\ne 0,\\ 0, &{}u=0.\\ \end{array}\right. } \end{aligned}$$

It is obvious that $\xi (u,v)>0$ if $u\ne 0$. $u\in {\mathcal {M}}$ if and only if $\xi (u^+,u^-)=\xi (u^-,u^+)=1$. Next we define

$$\begin{aligned} {\mathbf {U}}=\Bigl \{u\in H^1_r({\mathbb {R}}^3):|\xi (u^+,u^-)-1|<\frac{1}{2}, |\xi (u^-,u^+)-1|<\frac{1}{2}\Bigr \}. \end{aligned}$$

(3.1)

Lemma 3.1

There exists a sequence $\{u_n\}\subset {\mathbf {U}}$ such that $I(u_n)\rightarrow m$ and $I'(u_n)\rightarrow 0$.

Proof

We will break the proof into three claims.

Claim 1: $\inf _{u\in {\mathcal {M}}}I(u)=\inf _{\sigma \in \Sigma }\sup _{u\in \sigma (Q)}I(u)=m$.

We first recall that, for each $u\in {\mathcal {M}}$, there exists $\sigma (s,t)=\gamma t(1-s)u^++\gamma tsu^-\in \Sigma$ for $\gamma >0$ large enough. Therefore, from Lemma 2.2 we conclude that

$$\begin{aligned} I(u)=\max _{s,t\ge 0}I(su^++tu^-)\ge \sup _{u\in \sigma (Q)}I(u)\ge \inf _{\sigma \in \Sigma }\sup _{u\in \sigma (Q)}I(u), \end{aligned}$$

which means that

$$\begin{aligned} \inf _{u\in {\mathcal {M}}}I(u)\ge \inf _{\sigma \in \Sigma }\sup _{u\in \sigma (Q)}I(u). \end{aligned}$$

In what follows, we prove the opposite. Due to the facts that for each $\sigma \in \Sigma$ with $t\in [0,1]$, $\sigma (0,t)\in P$ and $\sigma (1,t)\in -P$, it holds that

$$\begin{aligned} \xi (\sigma ^+(0,t),\sigma ^-(0,t))-\xi (\sigma ^-(0,t),\sigma ^+(0,t))=\xi (\sigma ^+(0,t),\sigma ^-(0,t))\ge 0 \end{aligned}$$

(3.2)

and

$$\begin{aligned} \xi (\sigma ^+(1,t),\sigma ^-(1,t))-\xi (\sigma ^-(1,t),\sigma ^+(1,t))=-\xi (\sigma ^-(1,t),\sigma ^+(1,t))\le 0. \end{aligned}$$

(3.3)

Again from the definition of $\Sigma$, for all $\sigma \in \Sigma$ and $s\in [0,1]$, there exists

$$\begin{aligned} \begin{aligned}&\xi (\sigma ^+(s,1),\sigma ^-(s,1))+\xi (\sigma ^-(s,1),\sigma ^+(s,1))\\&\quad \ge \frac{\int _{{\mathbb {R}}^3}|\sigma (s,1)|^{p+1}dx+\int _{{\mathbb {R}}^3}|\sigma (s,1)|^6dx}{\Vert \sigma (s,1)\Vert ^2+L_{\phi _{\sigma (s,1)}}(\sigma (s,1))}\\&\quad \ge 2. \end{aligned} \end{aligned}$$

So, we have

$$\begin{aligned} \xi (\sigma ^+(s,1),\sigma ^-(s,1))+\xi (\sigma ^-(s,1),\sigma ^+(s,1))-2\ge 0 \end{aligned}$$

(3.4)

and

$$\begin{aligned} \xi (\sigma ^+(s,0),\sigma ^-(s,0))+\xi (\sigma ^-(s,0),\sigma ^+(s,0))-2=-2<0. \end{aligned}$$

(3.5)

On account of (3.2), (3.3), (3.4) and (3.5), it follows from Miranda’s theorem [22] that there exists $(s_\sigma ,t_\sigma )\in Q$ such that

$$\begin{aligned} \begin{aligned} 0&=\xi (\sigma ^+(s_\sigma ,t_\sigma ),\sigma ^-(s_\sigma ,t_\sigma ))-\xi (\sigma ^-(s_\sigma ,t_\sigma ),\sigma ^+(s_\sigma ,t_\sigma ))\\&=\xi (\sigma ^+(s_\sigma ,t_\sigma ),\sigma ^-(s_\sigma ,t_\sigma ))+\xi (\sigma ^-(s_\sigma ,t_\sigma ),\sigma ^+(s_\sigma ,t_\sigma ))-2, \end{aligned} \end{aligned}$$

which evidently gives

$$\begin{aligned} \xi (\sigma ^+(s_\sigma ,t_\sigma ),\sigma ^-(s_\sigma ,t_\sigma ))=\xi (\sigma ^-(s_\sigma ,t_\sigma ),\sigma ^+(s_\sigma ,t_\sigma ))=1. \end{aligned}$$

That is, for all $\sigma \in \Sigma$, there exists $u_\sigma =\sigma (s_\sigma ,t_\sigma )\in \sigma (Q)\cap {\mathcal {M}}$. Hence, we have

$$\begin{aligned} \sup _{u\in \sigma (Q)}I(u)\ge I(u_\sigma )\ge \inf _{u\in {\mathcal {M}}}I(u), \end{aligned}$$

namely

$$\begin{aligned} \inf _{\sigma \in \Sigma }\sup _{u\in \sigma (Q)}I(u)\ge \inf _{u\in {\mathcal {M}}}I(u). \end{aligned}$$

Claim 2: There is a (PS)$_m$-sequence $\{u_n\}$ in $H^1_r({\mathbb {R}}^3)$ for I.

With the previous in our mind, we now consider a minimizing sequence $\omega _n\in {\mathcal {M}}$ and $\sigma _n(s,t)\in \Sigma$. Then, by Claim 1, it gives

$$\begin{aligned} \lim _{n\rightarrow \infty }\max _{\omega \in \sigma _n(Q)}I(\omega )=\lim _{n\rightarrow \infty }I(\omega _n). \end{aligned}$$

Using a variant form of the classical deformation lemma [23] due to Hofer [14], we claim that there exists $\{u_n\}\subset H^1_r({\mathbb {R}}^3)$ such that

$$\begin{aligned} I(u_n)\rightarrow m,\quad I'(u_n)\rightarrow 0 \quad \text{ and }\quad \text{ dist }(u_n,\sigma _n(Q))\rightarrow 0\quad \text{ as }\quad n\rightarrow \infty . \end{aligned}$$

(3.6)

Nevertheless, we will give a detailed proof below for the reader’s convenience. Suppose by contradiction, there exists $\delta >0$ such that $\sigma _n(Q)\cap {\mathbf {W}}_\delta =\emptyset$ for n large enough, where

$$\begin{aligned} {\mathbf {W}}_\delta =\{u\in E:\exists v\in H^1_r({\mathbb {R}}^3), s.t.\,\, \Vert v-u\Vert \le \delta ,\Vert I'(v)\Vert \le \delta , |I(v)-m|\le \delta \}. \end{aligned}$$

In light of Hofer [14, Lemma 1], there exists a continuous map $\eta :[0,1]\times H^1_r({\mathbb {R}}^3)$ satisfying for some $\epsilon \in (0,m/2)$ and all $t\in [0,1]$,

(i):

$\eta (0,u)=u$, $\eta (t,-u)=-\eta (t,u)$,

(ii):

$\eta (t,u)=u$, $\forall u\in I^{m-\epsilon }\cup (H^1_r({\mathbb {R}}^3)\setminus I^{m+\epsilon })$,

(iii):

$\eta (1,I^{m+\frac{\epsilon }{2}}\setminus {\mathbf {W}}_\delta )\subset I^{m-\frac{\epsilon }{2}}$,

(iv):

$\eta (1,(I^{m+\frac{\epsilon }{2}}\cap P)\setminus {\mathbf {W}}_\delta )\subset I^{m-\frac{\epsilon }{2}}\cap P$, where $I^d:=\{u\in H^1_r({\mathbb {R}}^3):I(u)\le d\}$.

Since

$$\begin{aligned} \lim _{n\rightarrow \infty }\max _{\omega \in \sigma _n(Q)}I(\omega )=\lim _{n\rightarrow \infty }I(\omega _n)=m, \end{aligned}$$

we can choose n large enough such that

$$\begin{aligned} \sigma _n(Q)\subset I^{m+\frac{\epsilon }{2}}\quad \text{ and } \quad \sigma _n(Q)\cap {\mathbf {W}}_\delta =\emptyset . \end{aligned}$$

(3.7)

Denoting by ${\widetilde{\sigma }}_n(s,t)=\eta (1,\sigma _n(s,t)),\ \forall (s,t)\in Q$, we declare that ${\widetilde{\sigma }}_n\in \Sigma$. Thus ${\widetilde{\sigma }}_n(Q)\subset I^{m-\frac{\epsilon }{2}}$ by using (3.7) and property (iii) of $\eta$, which leads to a contradiction, since

$$\begin{aligned} m=\inf _{\sigma \in \Sigma }\sup _{\omega \in \sigma (Q)}I(\omega )\le \max _{\omega \in {\widetilde{\sigma }}_n(Q)}I(\omega )\le m-\frac{\epsilon }{2}. \end{aligned}$$

Actually, since $\sigma _n\in \Sigma$, it follows from (ii) that ${\widetilde{\sigma }}_n(s,0)=\eta (1,\sigma _n(s,0))=\eta (1,0)=0$. On the one hand, recalling from (3.7), $\sigma _n(0,t)\in P$ and (iv), we see that ${\widetilde{\sigma }}_n(0,t)\in P$. On the other hand, due to (3.7) and $\sigma _n(1,t)\in -P$, we have $-\sigma _n(1,t)\in (I^{m+\frac{\epsilon }{2}}\cap P)\setminus {\mathbf {W}}_\delta$, which implies from (i) and (iv) that ${\widetilde{\sigma }}_n(1,t)=\eta (1,\sigma _n(1,t))=-\eta (1,-\sigma _n(1,t))\in -P$. Then ${\widetilde{\sigma }}_n$ satisfies property (a). Moreover, using the fact that $(I\circ \sigma _n)(s,1)\le 0$, we deduce from (ii) that ${\widetilde{\sigma }}_n(s,1)=\eta (1,\sigma _n(s,1))=\sigma _n(s,1)$, thus ${\widetilde{\sigma }}_n$ satisfies property (b). Therefore, upon the continuity of $\eta$ and $\sigma _n$, we deduce that ${\widetilde{\sigma }}_n\in \Sigma$.

Claim 3: The sequence $\{u_n\}$ in Claim 2 satisfies $\{u_n\}\subset {\mathbf {U}}$ for n large enough.

We note that

$$\begin{aligned} \begin{aligned} m+\Vert u_n\Vert +o(1)&\ge I(u_n)=I(u_n)-\frac{1}{p+1}\langle I'(u_n),u_n\rangle \\&=\Bigl (\frac{1}{2}-\frac{1}{p+1}\Bigr )\Vert u_n\Vert ^2+\Bigl (\frac{1}{4}-\frac{1}{p+1}\Bigr )L_{\phi _{u_n}}(u_n) -\Bigl (\frac{1}{6}-\frac{1}{p+1}\Bigr )\int _{{\mathbb {R}}^3}|u_n|^6dx\\&\ge \Bigl (\frac{1}{2}-\frac{1}{p+1}\Bigr )\Vert u_n\Vert ^2, \end{aligned} \end{aligned}$$

which means $\{u_n\}$ is bounded. Then we have $\langle I'(u_n),u_n^\pm \rangle =o(1)$ since $I'(u_n)\rightarrow 0$. Therefore, to reach the claim, it suffices to show that $u_n^\pm \ne 0$, which means that $\xi (u_n^+,u_n^-)\rightarrow 1$, $\xi (u_n^-,u_n^+)\rightarrow 1$ and then $\{u_n\}\subset {\mathbf {U}}$ for n large enough. From (3.6), there exists a sequence $\{v_n\}$ such that

$$\begin{aligned} v_n=\alpha _n\omega _n^++\beta _n\omega _n^-\in \sigma _n(Q)\quad \text{ and }\quad \Vert v_n-u_n\Vert \rightarrow 0. \end{aligned}$$

(3.8)

We now prove that $\alpha _n\omega _n^+\ne 0$ and $\beta _n\omega _n^-\ne 0$ for n large enough, so it indicates that $u_n^\pm \ne 0$. Since $\omega _n\in {\mathcal {M}}$, using Lemma 2.3 (iii), we only need to show that $\alpha _n\nrightarrow 0$ and $\beta _n\nrightarrow 0$ for n large enough. Suppose by contradiction that $\alpha _n\rightarrow 0$, by the continuity of I and (3.8) one has that

$$\begin{aligned} 0<m=\lim _{n\rightarrow \infty }I(v_n)=\lim _{n\rightarrow \infty }I(\alpha _n\omega _n^++\beta _n\omega _n^-) =\lim _{n\rightarrow \infty }I(\beta _n\omega _n^-). \end{aligned}$$

Then, by Lemma 2.2 and $\Lambda _1\le \Vert \omega _n^+\Vert \le \Lambda _2$, we see that

$$\begin{aligned} \begin{aligned} m&=\lim _{n\rightarrow \infty }I(\omega _n)=\lim _{n\rightarrow \infty }\max _{s,t\ge 0}I(s\omega _n^++t\omega _n^-)\\&\ge \lim _{n\rightarrow \infty }\max _{s\ge 0}I(s\omega _n^++\beta _n\omega _n^-)\\&=\lim _{n\rightarrow \infty }\max _{s\ge 0}\Bigl [\frac{s^2}{2}\Vert \omega _n^+\Vert ^2+\frac{s^4}{4} L_{\phi _{\omega _n^+}}(\omega _n^+) -\frac{s^{p+1}}{p+1}\int _{{\mathbb {R}}^3}|\omega _n^+|^{p+1}dx-\frac{s^6}{6}\int _{{\mathbb {R}}^3}|\omega _n^+|^{6}dx\\&\ \ \ +\frac{s^2\beta _n^2}{2} L_{\phi _{\omega _n^+}}(\omega _n^-)+I(\beta _n\omega _n^-)\Bigr ]\\&\ge \lim _{n\rightarrow \infty }\max _{s\ge 0}\Bigl [\frac{s^2}{2}\Vert \omega _n^+\Vert ^2-C(s^{p+1}\Vert \omega _n^+\Vert ^{p+1}+s^6\Vert \omega _n^+\Vert ^6)\Bigr ]+m\\&\ge \max _{s\ge 0}\Bigl [\frac{Cs^2}{2}-C(s^{p+1}+s^6)\Bigr ]+m>m, \end{aligned} \end{aligned}$$

which reaches to a contradiction. This concludes the proof. $\square$

Lemma 3.2

Any bounded $\{u_n\}\subset {\mathbf {U}}\subset H^1_r({\mathbb {R}}^3)$ such that $I(u_n)\rightarrow b\in (0,c+\frac{1}{3}S^{\frac{3}{2}})$ and $I'(u_n)\rightarrow 0$ contains a convergent subsequence.

Proof

Since $\{u_n\}$ is bounded in $H^1_r({\mathbb {R}}^3)$, there exists $u\in H^1_r({\mathbb {R}}^3)$ such that, up to a subsequence,

$$\begin{aligned} u_n\rightharpoonup u\ \text{ in }\ H^1_r({\mathbb {R}}^3),\quad u_n \rightarrow u \ \text{ in }\ L^{p+1}({\mathbb {R}}^3),\quad u_n\rightarrow u\ \text{ a.e. }\ \text{ in }\ {\mathbb {R}}^3. \end{aligned}$$

First we show that $I'(u)=0$. In fact, let us define $w_n=|u_n|^4 u_n$ and $w=|u|^4 u$. Since $\{u_n\}$ is bounded in $L^6({\mathbb {R}}^3)$, then $w_n$ is bounded in $L^{\frac{6}{5}}({\mathbb {R}}^3)$ and so $w_n\rightharpoonup w$ in $L^{\frac{6}{5}}({\mathbb {R}}^3)$. According to the fact that for any $v\in H^1_r({\mathbb {R}}^3)$, one has $v\in L^6({\mathbb {R}}^3)$. Therefore, it holds that

$$\begin{aligned} \int _{{\mathbb {R}}^3}w_nvdx\rightarrow \int _{{\mathbb {R}}^3}wvdx,\ \text{ i.e. },\ \int _{{\mathbb {R}}^3}|u_n|^4 u_n vdx\rightarrow \int _{{\mathbb {R}}^3}|u|^4 u vdx, \end{aligned}$$

(3.9)

and

$$\begin{aligned} \int _{{\mathbb {R}}^3}(\nabla u_n\cdot \nabla v+u_nv)dx\rightarrow \int _{{\mathbb {R}}^3}(\nabla u\cdot \nabla v+uv)dx. \end{aligned}$$

(3.10)

Similar to (3.9), we can also conclude that

$$\begin{aligned} \int _{{\mathbb {R}}^3}|u_n|^{p-1} u_n vdx\rightarrow \int _{{\mathbb {R}}^3}|u|^{p-1} u vdx\quad \text{ and } \quad \int _{{\mathbb {R}}^3}\phi _{u_n}u_nvdx\rightarrow \int _{{\mathbb {R}}^3}\phi _uuvdx. \end{aligned}$$

(3.11)

Combining (3.9)–(3.11), we derive that

$$\begin{aligned} \begin{aligned} \langle I'(u_n),v\rangle&=\int _{{\mathbb {R}}^3}(\nabla u_n\cdot \nabla v+u_nv)dx+\int _{{\mathbb {R}}^3}\phi _{u_n}u_nvdx\\&\ \ \ \ -\int _{{\mathbb {R}}^3}|u_n|^{p-1} u_n vdx-\int _{{\mathbb {R}}^3}|u_n|^4 u_n vdx\\&\rightarrow \int _{{\mathbb {R}}^3}(\nabla u\cdot \nabla v+uv)dx+\int _{{\mathbb {R}}^3}\phi _uuvdx\\&\ \ \ \ -\int _{{\mathbb {R}}^3}|u|^{p-1} u vdx-\int _{{\mathbb {R}}^3}|u|^4 u vdx=\langle I'(u),v\rangle . \end{aligned} \end{aligned}$$

Due to the fact that $I'(u_n)\rightarrow 0$ in $H^1_r({\mathbb {R}}^3)$, it therefore follows that $I'(u)=0$ in $H^1_r({\mathbb {R}}^3)$.

Denoting by ${\widetilde{u}}_n:=u_n-u$, from Brézis-Lieb lemma (see [7, Theorem 1]), Lemma 2.1 (iii) and the compactness of $H^1_r({\mathbb {R}}^3)\hookrightarrow L^s({\mathbb {R}}^3)$ for $s\in (2,6)$, we deduce that

$$\begin{aligned} \begin{aligned} b&=I(u_n)+o(1)\\&=\frac{1}{2}\Vert u_n\Vert ^2+\frac{1}{4}L_{\phi _{u_n}}(u_n) -\frac{1}{p+1}\int _{{\mathbb {R}}^3}|u_n|^{p+1}dx-\frac{1}{6}\int _{{\mathbb {R}}^3}|u_n|^{6}dx+o(1)\\&=\frac{1}{2}\Vert u\Vert ^2+\frac{1}{4}L_{\phi _{u}}(u)-\frac{1}{p+1}\int _{{\mathbb {R}}^3}|u|^{p+1}dx-\frac{1}{6}\int _{{\mathbb {R}}^3}|u|^{6}dx\\&\ \ \ +\frac{1}{2}\Vert {\widetilde{u}}_n\Vert ^2 -\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n|^{6}dx+o(1)\\&=I(u)+\frac{1}{2}\Vert {\widetilde{u}}_n\Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n|^{6}dx+o(1)\\&=I(u)+\Bigl (\frac{1}{2}\Vert {\widetilde{u}}_n^+\Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx\Bigr )+ \Bigl (\frac{1}{2}\Vert {\widetilde{u}}_n^-\Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^-|^{6}dx\Bigr )+o(1), \end{aligned} \end{aligned}$$

(3.12)

$$\begin{aligned} 0&=\langle I'(u_n),u_n^+\rangle +o(1)\nonumber \\&=\Vert u_n^+\Vert ^2+L_{\phi _{u_n}}(u_n^+) -\int _{{\mathbb {R}}^3}|u_n^+|^{p+1}dx-\int _{{\mathbb {R}}^3}|u_n^+|^{6}dx+o(1)\nonumber \\&=\Vert u^+\Vert ^2+ L_{\phi _{u}}(u^+)-\int _{{\mathbb {R}}^3}|u^+|^{p+1}dx-\int _{{\mathbb {R}}^3}|u^+|^{6}dx\nonumber \\&\ \ \ +\Vert {\widetilde{u}}_n^+\Vert ^2-\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx+o(1)\nonumber \\&=\langle I'(u),u^+\rangle +\Vert {\widetilde{u}}_n^+\Vert ^2-\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx+o(1)\nonumber \\&=\Vert {\widetilde{u}}_n^+\Vert ^2-\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx+o(1), \end{aligned}$$

(3.13)

and

$$\begin{aligned} \begin{aligned} 0&=\langle I'(u_n),u_n^-\rangle +o(1)\\&=\Vert u_n^-\Vert ^2+ L_{\phi _{u_n}}(u_n^-) -\int _{{\mathbb {R}}^3}|u_n^-|^{p+1}dx-\int _{{\mathbb {R}}^3}|u_n^-|^{6}dx+o(1)\\&=\Vert u^-\Vert ^2+ L_{\phi _{u}}(u^-)-\int _{{\mathbb {R}}^3}|u^-|^{p+1}dx-\int _{{\mathbb {R}}^3}|u^-|^{6}dx\\&\ \ \ +\Vert {\widetilde{u}}_n^-\Vert ^2-\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^-|^{6}dx+o(1)\\&=\langle I'(u),u^-\rangle +\Vert {\widetilde{u}}_n^-\Vert ^2-\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^-|^{6}dx+o(1)\\&=\Vert {\widetilde{u}}_n^-\Vert ^2-\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^-|^{6}dx+o(1). \end{aligned} \end{aligned}$$

(3.14)

In what follows, we show that $u_n\rightarrow u$ in $H^1_r({\mathbb {R}}^3)$ by excluding the other three possibilities.

Case 1. ${\widetilde{u}}_n^+\rightharpoonup 0$ and ${\widetilde{u}}_n^-\rightarrow 0$ in $H^1_r({\mathbb {R}}^3)$.

From ${\widetilde{u}}_n^-\rightarrow 0$ strongly in $H^1_r({\mathbb {R}}^3)$, it gives $u_n^-\rightarrow u^-$ in $H^1_r({\mathbb {R}}^3)$. Then, for $u_n\in {\mathbf {U}}$, by the definition of $\xi (u,v)$, we see that

$$\begin{aligned} \begin{aligned} \frac{1}{2}\Vert u_n^-\Vert ^2&<\frac{1}{2}\Bigl (\Vert u_n^-\Vert ^2+L_{\phi _{u_n}}(u_n^-)\Bigr )\\&<\int _{{\mathbb {R}}^3}|u_n^-|^{p+1}dx+\int _{{\mathbb {R}}^3}|u_n^-|^{6}dx\\&\le C(\Vert u_n^-\Vert ^{p+1}+\Vert u_n^-\Vert ^6). \end{aligned} \end{aligned}$$

Thus, there exists $\varrho >0$ such that $\Vert u_n^-\Vert \ge \varrho >0$ for any $u_n\in {\mathbf {U}}$. It therefore follows that $\Vert u^-\Vert ^2=\lim _{n\rightarrow \infty }\Vert u_n^-\Vert ^2\ge \varrho ^2>0$, which means that $u\ne 0$. Since $I'(u)=0$, then $u\in {\mathcal {N}}$ and $I(u)\ge c$.

Moreover, since ${\widetilde{u}}_n^+\rightharpoonup 0$ in $H^1_r({\mathbb {R}}^3)$, we may assume that for n large enough, $\Vert {\widetilde{u}}_n^+\Vert ^2\rightarrow d>0$. Then, by (3.13) we have $\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx\rightarrow d>0$. It follows from (1.7) that $\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx\le S^{-3}\Vert {\widetilde{u}}_n^+\Vert ^6$. So, it gives that $d\ge S^{\frac{3}{2}}$ and

$$\begin{aligned} \frac{1}{2}\Vert {\widetilde{u}}_n^+\Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx\rightarrow \frac{1}{3}d\ge \frac{1}{3}S^{\frac{3}{2}}. \end{aligned}$$

Noting that ${\widetilde{u}}_n^-\rightarrow 0$ in $H^1_r({\mathbb {R}}^3)$, we deduce from (3.12) that

$$\begin{aligned} b+o(1)=I(u_n)=I(u)+\Bigl (\frac{1}{2}\Vert {\widetilde{u}}_n^+\Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx\Bigr )\ge c+\frac{1}{3}S^{\frac{3}{2}}+o(1), \end{aligned}$$

which contradicts the fact that $b\in (0,c+\frac{1}{3}S^{\frac{3}{2}})$.

Case 2. ${\widetilde{u}}_n^-\rightharpoonup 0$ and ${\widetilde{u}}_n^+\rightarrow 0$ in $H^1_r({\mathbb {R}}^3)$.

From a similar argument as in Case 1, it can also lead to a contradiction.

Case 3. ${\widetilde{u}}_n^\pm \rightharpoonup 0$ in $H^1_r({\mathbb {R}}^3)$.

In the same way as Case 1 and Case 2, one has from (3.13) and (3.14) that

$$\begin{aligned} \frac{1}{2}\Vert {\widetilde{u}}_n^+\Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx\rightarrow \frac{1}{3}d\ge \frac{1}{3}S^{\frac{3}{2}} \end{aligned}$$

and

$$\begin{aligned} \frac{1}{2}\Vert {\widetilde{u}}_n^-\Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^-|^{6}dx\rightarrow \frac{1}{3}d\ge \frac{1}{3}S^{\frac{3}{2}}. \end{aligned}$$

Note that $I(u)=I(u)-\frac{1}{p+1}\langle I'(u),u\rangle \ge \Bigl (\frac{1}{2}-\frac{1}{p+1}\Bigr )\Vert u\Vert ^2\ge 0$, it can be concluded from Lemma 2.4 (ii) that

$$\begin{aligned} \begin{aligned} b&=I(u_n)+o(1)\\&=I(u)+\Bigl (\frac{1}{2}\Vert {\widetilde{u}}_n^+\Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^+|^{6}dx\Bigr ) +\Bigl (\frac{1}{2}\Vert {\widetilde{u}}_n^-\Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|{\widetilde{u}}_n^-|^{6}dx\Bigr )+o(1)\\&\ge \frac{1}{3}S^{\frac{3}{2}}+\frac{1}{3}S^{\frac{3}{2}}+o(1)>c+\frac{1}{3}S^{\frac{3}{2}}+o(1), \end{aligned} \end{aligned}$$

which is impossible since $b\in (0,c+\frac{1}{3}S^{\frac{3}{2}})$. $\square$

Lemma 3.3

$m<c+\frac{1}{3}S^{\frac{3}{2}}$.

Proof

The idea here is to find an element in ${\mathcal {M}}$ such that the value of I is strictly less than $c+\frac{1}{3}S^{\frac{3}{2}}$ on this element. For this purpose, we need the extremal function $u_{\varepsilon ,y}$ defined by

$$\begin{aligned} u_{\varepsilon ,y}=\frac{(3\varepsilon )^{\frac{1}{4}}}{\Bigl (\varepsilon +|x-y|^2\Bigr )^{\frac{1}{2}}},\ \ \ \forall \varepsilon >0,\ y\in {\mathbb {R}}^3. \end{aligned}$$

Let $\varphi \in C_0^{\infty }(B_{2\rho }(0))$ such that $0\le \varphi \le 1$, $\varphi \equiv 1$ on $B_{\rho }(0)$ and $\text{ supp }(\varphi ) \subset B_{2\rho }(0)$ for some $\rho >0$. Set $v_\varepsilon =\varphi \circ u_{\varepsilon ,0}$ and then $v_\varepsilon \in H^1_r({\mathbb {R}}^3)$ with $v_\varepsilon (x)\ge 0$ for each $x\in {\mathbb {R}}^3$. According to the argument in [8, Lemma 1.1], we have the following asymptotic estimates

$$\begin{aligned}&\int _{{\mathbb {R}}^3}|\nabla v_\varepsilon |^2dx=k_1+O(\varepsilon ^{\frac{1}{2}}),\ \ \ \ \Bigl (\int _{{\mathbb {R}}^3}|v_\varepsilon |^{6}dx\Bigr )^{\frac{1}{3}}=k_2+O(\varepsilon ^{\frac{3}{2}}),\ \ \ \ \frac{k_1}{k_2}=S, \end{aligned}$$

(3.15)

$$\begin{aligned}&\int _{{\mathbb {R}}^3}|v_\varepsilon |^{s}dx= {\left\{ \begin{array}{ll} O(\varepsilon ^{\frac{s}{4}}), &{}s\in [2,3),\\ O(\varepsilon ^{\frac{s}{4}}|\ln \varepsilon |), &{}s=3,\\ O(\varepsilon ^{\frac{6-s}{4}}), &{}s\in (3,6). \end{array}\right. } \end{aligned}$$

(3.16)

Suppose that $u_0(x)$ is the positive ground state solution of system (1.1) obtained in Lemma 2.4. We first prove that there exist $s_\varepsilon , t_\varepsilon >0$ such that $s_\varepsilon u_0-t_\varepsilon v_\varepsilon \in {\mathcal {M}}$. Actually, denote $\psi (\tau )=\frac{1}{\tau }u_0-v_\varepsilon$, $\tau >0$ and define $\tau _1, \tau _2\in (0,\infty ]$ by

$$\begin{aligned} \tau _1=\sup \{\tau \in {\mathbb {R}}^+:\psi ^+(\tau )\ne 0\},\quad \tau _2=\inf \{\tau \in {\mathbb {R}}^+:\psi ^-(\tau )\ne 0\}. \end{aligned}$$

In view of the positivity and regularity of $u_0$, we have $\tau _1=\infty$ and $0<\tau _2<\tau _1$. If $\tau \rightarrow \tau _2^+$, this immediately implies $\psi ^-(\tau )\rightarrow 0$ and $\psi ^+(\tau )\rightarrow \frac{1}{\tau _2}u_0-v_\varepsilon \ne 0$. Thus we use Lemma 2.3 (ii) to obtain that $t_{\psi (\tau )}\rightarrow \infty$ as $\tau \rightarrow \tau _2^+$. Furthermore, since $\psi ^+(\tau )\rightarrow \frac{1}{\tau _2}u_0-v_\varepsilon \ne 0$ in $H^1_r({\mathbb {R}}^3)$ and

$$\begin{aligned} \begin{aligned} 0&=\frac{\langle I'(s_{\psi (\tau )}\psi ^+(\tau )+t_{\psi (\tau )}\psi ^-(\tau )), s_{\psi (\tau )}\psi ^+(\tau ) \rangle }{s^4_{\psi (\tau )}}\\&=\frac{\Vert \psi ^+(\tau )\Vert ^2}{s^2_{\psi (\tau )}}+ L_{\phi _{\psi ^+(\tau )}}(\psi ^+(\tau ))+\frac{t^2_{\psi (\tau )}}{s^2_{\psi (\tau )}}L_{\phi _{\psi ^-(\tau )}}(\psi ^+(\tau ))\\&\ \ \ -s^{p-3}_{\psi (\tau )}\int _{{\mathbb {R}}^3}|\psi ^+(\tau )|^pdx-s^2_{\psi (\tau )}\int _{{\mathbb {R}}^3}|\psi ^+(\tau )|^6dx, \end{aligned} \end{aligned}$$

it is evident to see that $\frac{t_{\psi (\tau )}}{s_{\psi (\tau )}}$ is unbounded, and so

$$\begin{aligned} s_{\psi (\tau )}-t_{\psi (\tau )}\rightarrow -\infty \quad \text{ as } \quad \tau \rightarrow \tau _2^+. \end{aligned}$$

If $\tau \rightarrow \tau _1=\infty$, following a similar argument as above, we can derive that $\frac{s_{\psi (\tau )}}{t_{\psi (\tau )}}$ is unbounded and

$$\begin{aligned} s_{\psi (\tau )}-t_{\psi (\tau )}\rightarrow \infty \quad \text{ as } \quad \tau \rightarrow \tau _1. \end{aligned}$$

Then, thanks to the continuity of s and t, there exists $\tau _\varepsilon \in (\tau _2,\tau _1)$ such that $s_{\psi (\tau _\varepsilon )}=t_{\psi (\tau _\varepsilon )}$. Let $s_\varepsilon =\frac{1}{\tau _\varepsilon }s_{\psi (\tau _\varepsilon )}$ and $t_\varepsilon =t_{\psi (\tau _\varepsilon )}$, it is obvious that

$$\begin{aligned} s_{\psi (\tau _\varepsilon )}\psi ^+(\tau _\varepsilon )+t_{\psi (\tau _\varepsilon )}\psi ^-(\tau _\varepsilon ) =s_\varepsilon u_0-t_\varepsilon v_\varepsilon \in {\mathcal {M}}. \end{aligned}$$

Furthermore, it follows from Lemma 2.2 that $I(s_\varepsilon u_0-t_\varepsilon v_\varepsilon )=\sup _{s,t\ge 0}I(su_0-tv_\varepsilon )$.

Secondly, we shall show that $\sup _{s,t\ge 0}I(su_0-tv_\varepsilon )<c+\frac{1}{3}S^{\frac{3}{2}}$. A direct calculation implies that $I(su_0-tv_\varepsilon )<0$ as s or t large enough. In addition, the continuity of I respect to t also means that $I(su_0-tv_\varepsilon )<c+\frac{1}{3}S^{\frac{3}{2}}$ for t small enough. So we just need to consider the case that s and t are contained in some bounded domain such that t has a positive lower bound. For this case, we analyse the energy functional as follows. A straightforward calculation gives that

$$\begin{aligned} \begin{aligned} I(su_0-tv_\varepsilon )&=I(su_0)+\frac{1}{2}\Vert tv_\varepsilon \Vert ^2-\frac{1}{p+1}\int _{{\mathbb {R}}^3}|tv_\varepsilon |^{p+1}dx -\frac{1}{6}\int _{{\mathbb {R}}^3}|tv_\varepsilon |^{6}dx\\&\ \ \ +\frac{1}{4}\Bigl [L_{\phi _{su_0-tv_\varepsilon }}(su_0-tv_\varepsilon )-L_{\phi _{su_0}}(su_0)\Bigr ]\\&\ \ \ -\frac{1}{p+1}\int _{{\mathbb {R}}^3}\Bigl (|su_0-tv_\varepsilon |^{p+1}-|su_0|^{p+1}-|tv_\varepsilon |^{p+1}\Bigr )dx\\&\ \ \ -\frac{1}{6}\int _{{\mathbb {R}}^3}\Bigl (|su_0-tv_\varepsilon |^6-|su_0|^6-|tv_\varepsilon |^6\Bigr )dx\\&\ \ \ -st\int _{{\mathbb {R}}^3}(\nabla u_0\cdot \nabla v_\varepsilon +u_0v_\varepsilon )dx. \end{aligned} \end{aligned}$$

(3.17)

In light of the estimates (3.15) and (3.16), we deduce that as $\varepsilon \rightarrow 0$,

$$\begin{aligned} \max _{t\ge 0}\Bigl (\frac{1}{2}\Vert tv_\varepsilon \Vert ^2-\frac{1}{6}\int _{{\mathbb {R}}^3}|tv_\varepsilon |^6dx\Bigr ) =\frac{1}{3}\frac{\Vert v_\varepsilon \Vert ^3}{\Bigl (\int _{{\mathbb {R}}^3}|v_\varepsilon |^6dx\Bigr )^{\frac{1}{2}}} =\frac{1}{3}S^{\frac{3}{2}}+O(\varepsilon ^{\frac{1}{2}}). \end{aligned}$$

(3.18)

By the definition of $v_\varepsilon$, if $\varepsilon \in (0,\rho ^2]$, we have

$$\begin{aligned} \begin{aligned} \int _{{\mathbb {R}}^3}|tv_\varepsilon |^{p+1}dx&\ge C\int _{|x|\le \rho }|\varphi \circ u_{\varepsilon ,0}|^{p+1}dx=C\int _{|x|\le \rho }\Bigl |\frac{(3\varepsilon )^{\frac{1}{4}}}{(\varepsilon +|x|^2)^{\frac{1}{2}}}\Bigr |^{p+1}dx\\&=C\varepsilon ^{\frac{p+1}{4}}\int _0^\rho \frac{r^2}{(\varepsilon +r^2)^{\frac{p+1}{2}}}dr =C\varepsilon ^{\frac{5-p}{4}}\int _0^{\rho \varepsilon ^{-\frac{1}{2}}}\frac{\mu ^2}{(1+\mu ^2)^{\frac{p+1}{2}}}d\mu \\&\ge C\varepsilon ^{\frac{5-p}{4}}\int _0^1\frac{\mu ^2}{2^{\frac{p+1}{2}}}d\mu =C\varepsilon ^{\frac{5-p}{4}}. \end{aligned} \end{aligned}$$

(3.19)

Since $u_0$ is a positive ground state solution of system (1.1), we are led to

$$\begin{aligned} \begin{aligned}&-st\int _{{\mathbb {R}}^3}(\nabla u_0\cdot \nabla v_\varepsilon +u_0v_\varepsilon )dx\\&=st\int _{{\mathbb {R}}^3}\Bigl [\phi _{u_0}u_0v_\varepsilon -|u_0|^{p-1}u_0v_\varepsilon -|u_0|^4u_0v_\varepsilon \Bigr ]\\&\le st\int _{{\mathbb {R}}^3}\phi _{u_0}u_0v_\varepsilon \le st\Vert \phi _{u_0}\Vert _6\Vert u_0\Vert _{\frac{12}{5}}\Vert v_\varepsilon \Vert _{\frac{12}{5}}\\&\le C\Vert v_\varepsilon \Vert _{\frac{12}{5}}\le C\varepsilon ^{\frac{1}{4}}. \end{aligned} \end{aligned}$$

(3.20)

Through a simple calculation, we see that

$$\begin{aligned} \begin{aligned}&\frac{1}{4}\Bigl [L_{\phi _{su_0-tv_\varepsilon }}(su_0-tv_\varepsilon )-L_{\phi _{su_0}}(su_0)\Bigr ]\\&\quad =-st\Bigl (\int _{{\mathbb {R}}^3}\phi _{su_0}u_0v_\varepsilon dx+\int _{{\mathbb {R}}^3}\phi _{tv_\varepsilon }u_0v_\varepsilon dx\Bigr )+\frac{t^2s^2}{2}\int _{{\mathbb {R}}^3}\phi _{u_0}v_\varepsilon ^2dx\\&\qquad +\frac{t^4}{4}\int _{{\mathbb {R}}^3}\phi _{v_\varepsilon }v_\varepsilon ^2dx +s^2t^2\int _{{\mathbb {R}}^3}\int _{{\mathbb {R}}^3}\frac{u_0(x)u_0(y)v_\varepsilon (x)v_\varepsilon (y)}{|x-y|}dxdy. \end{aligned} \end{aligned}$$

(3.21)

Note that s and t are bounded, we can infer from Hölder’s inequality, Lemma 2.1 (iv) and (3.16) that

$$\begin{aligned}&st\int _{{\mathbb {R}}^3}\phi _{su_0}u_0v_\varepsilon dx\le st\Vert \phi _{su_0}\Vert _6\Vert u_0\Vert _{\frac{12}{5}}\Vert v_\varepsilon \Vert _{\frac{12}{5}}\le C\Vert v_\varepsilon \Vert _{\frac{12}{5}}\le C\varepsilon ^{\frac{1}{4}}, \end{aligned}$$

(3.22)

$$\begin{aligned}&st\int _{{\mathbb {R}}^3}\phi _{tv_\varepsilon }u_0v_\varepsilon dx\le st\Vert \phi _{tv_\varepsilon }\Vert _6\Vert u_0\Vert _{\frac{12}{5}}\Vert v_\varepsilon \Vert _{\frac{12}{5}}\le C\Vert v_\varepsilon \Vert _{\frac{12}{5}}^3\le C\varepsilon ^{\frac{3}{4}}, \end{aligned}$$

(3.23)

$$\begin{aligned}&\frac{t^2s^2}{2}\int _{{\mathbb {R}}^3}\phi _{u_0}v_\varepsilon ^2dx\le C \Vert \phi _{u_0}\Vert _6\Vert v_\varepsilon \Vert ^2_{\frac{12}{5}} \le C\varepsilon ^{\frac{1}{2}}, \end{aligned}$$

(3.24)

$$\begin{aligned}&\frac{t^4}{4}\int _{{\mathbb {R}}^3}\phi _{v_\varepsilon }v_\varepsilon ^2dx\le C \Vert \phi _{v_\varepsilon }\Vert _6\Vert v_\varepsilon \Vert ^2_{\frac{12}{5}}\le C\Vert v_\varepsilon \Vert ^4_{\frac{12}{5}} \le C\varepsilon , \end{aligned}$$

(3.25)

and, by Hardy-Littlewood-Sobolev inequality, we get

$$\begin{aligned} \begin{aligned}&s^2t^2\int _{{\mathbb {R}}^3}\int _{{\mathbb {R}}^3}\frac{u_0(x)u_0(y)v_\varepsilon (x)v_\varepsilon (y)}{|x-y|}dxdy\\&\quad \le s^2t^2\Bigl (\int _{{\mathbb {R}}^3}|u_0(x)v_\varepsilon (x)|^{\frac{6}{5}}dx\Bigr )^{\frac{5}{3}}\le C\Vert u_0\Vert _{\frac{12}{5}}^2\Vert v_\varepsilon \Vert _{\frac{12}{5}}^2\le C\varepsilon ^{\frac{1}{2}}. \end{aligned} \end{aligned}$$

(3.26)

Then, in view of (3.22)–(3.26), we deduce that

$$\begin{aligned} \begin{aligned} \frac{1}{4}\Bigl [L_{\phi _{su_0-tv_\varepsilon }}(su_0-tv_\varepsilon )-L_{\phi _{su_0}}(su_0)\Bigr ]\le C\varepsilon ^{\frac{1}{4}}. \end{aligned} \end{aligned}$$

(3.27)

To proceed further, we need the inequality $|x-y|^q-x^q-y^q\ge -C(x^{q-1}y+xy^{q-1})$ for all $x,y\ge 0$ and $q\ge 1$ (see [28, Calculus Lemma]). This, combining with $u_0\in L^\infty ({\mathbb {R}}^3)$, $\text{ supp }(v_\varepsilon )\subset B_{2\rho }(0)$, Hölder’s inequality, (3.16) and the boundedness of s, t, readily shows that

$$\begin{aligned} \begin{aligned}&-\frac{1}{p+1}\int _{{\mathbb {R}}^3}\Bigl (|su_0-tv_\varepsilon |^{p+1}-|su_0|^{p+1}-|tv_\varepsilon |^{p+1}\Bigr )dx\\&\quad \le C\int _{{\mathbb {R}}^3}\Bigl (|su_0|^p|tv_\varepsilon |+|su_0||tv_\varepsilon |^p\Bigr )dx\\&\quad \le C\varepsilon ^{\frac{1}{4}}+C\Vert v_\varepsilon \Vert _p^p\le C\varepsilon ^{\frac{1}{4}} \end{aligned} \end{aligned}$$

(3.28)

and

$$\begin{aligned} \begin{aligned}&-\frac{1}{6}\int _{{\mathbb {R}}^3}\Bigl (|su_0-tv_\varepsilon |^6-|su_0|^6-|tv_\varepsilon |^6\Bigr )dx\\&\quad \le C\int _{{\mathbb {R}}^3}\Bigl (|su_0|^5|tv_\varepsilon |+|su_0||tv_\varepsilon |^5\Bigr )dx\\&\quad \le C\varepsilon ^{\frac{1}{4}}+C\Vert v_\varepsilon \Vert _5^5\le C\varepsilon ^{\frac{1}{4}}. \end{aligned} \end{aligned}$$

(3.29)

Substituting (3.18), (3.19), (3.20), (3.27), (3.28) and (3.29) into (3.17), we obtain from Lemma 2.4 and $4<p<5$ that

$$\begin{aligned} m\le I(su_0-tv_\varepsilon )\le I(u_0)+\frac{1}{3}S^{\frac{3}{2}}+C\varepsilon ^{\frac{1}{4}}-C \varepsilon ^{\frac{5-p}{4}}<c+\frac{1}{3}S^{\frac{3}{2}} \end{aligned}$$

(3.30)

as $\varepsilon \rightarrow 0$. That is, we complete the proof of this lemma. $\square$

Proof of Theorem 1.1

Based on Lemma 3.1, there exists a sequence $\{u_n\}\in {\mathbf {U}}$ such that $I(u_n)\rightarrow m$ and $I'(u_n)\rightarrow 0$ as $n\rightarrow \infty$. Then, Lemmas 3.2 and 3.3 indicate that there exists a nontrivial ${\widetilde{u}}_0\in H^1_r({\mathbb {R}}^3)$ such that $u_n\rightarrow {\widetilde{u}}_0$ as $n\rightarrow \infty$. Thus, by the continuity of I and $I'$, it gives that $I({\widetilde{u}}_0)=m$ and $I'({\widetilde{u}}_0)=0$. Moreover, similar to the proofs of Claim 1 and Claim 2 in Lemma 3.2, we can derive that $\Vert {\widetilde{u}}_0^\pm \Vert >0$, which immediately implies that ${\widetilde{u}}_0$ is a ground state sign-changing solution to system (1.1).

In what follows, we claim that ${\widetilde{u}}_0$ has exactly two nodal domains. We shall prove this by a contradiction argument. Supposing otherwise, then we assume that ${\widetilde{u}}_0=u_1+u_2+u_3$ with

$$\begin{aligned} u_i\ne 0,\quad u_1\ge 0,\quad u_2\le 0,\quad u_3\ge 0 \end{aligned}$$

and

$$\begin{aligned} \text{ supp }(u_i)\cap \text{ supp }(u_j)=\emptyset ,\quad i\ne j\ (i,j=1,2,3). \end{aligned}$$

By a direct calculation, we have

$$\begin{aligned} L_{\phi _{{\widetilde{u}}_0}}({\widetilde{u}}_0)=L_{\phi _{u_1}}(u_1)+L_{\phi _{u_2}}(u_2)+L_{\phi _{u_3}}(u_3)+2L_{\phi _{u_1}}(u_2)+2L_{\phi _{u_1}}(u_3)+2L_{\phi _{u_2}}(u_3). \end{aligned}$$

Let $v=u_1+u_2$, then it follows from $I'({\widetilde{u}}_0)=0$ that

$$\begin{aligned} \left\{ \begin{array}{ll} \langle I'(v),v^+\rangle =\langle I'(u_1+u_2),u_1\rangle =\langle I'({\widetilde{u}}_0),u_1\rangle - L_{\phi _{u_3}}(u_1)<0,\\ \langle I'(v),v^-\rangle =\langle I'(u_1+u_2),u_2\rangle =\langle I'({\widetilde{u}}_0),u_2\rangle - L_{\phi _{u_3}}(u_2)<0. \end{array} \right. \end{aligned}$$

Therefore, by Lemma 2.2, there exists $(s_v,t_v)\in (0,1]\times (0,1]$ such that

$$\begin{aligned} s_vv^++t_vv^-=s_vu_1+t_vu_2\in {\mathcal {M}}\quad \text{ and }\quad I(s_vu_1+t_vu_2)\ge m. \end{aligned}$$

Since $\langle I'({\widetilde{u}}_0),{\widetilde{u}}_0\rangle =0$ and $\langle I'(s_vu_1+t_vu_2),s_vu_1+t_vu_2\rangle =0$, we are led to

$$\begin{aligned} m&=I({\widetilde{u}}_0)-\frac{1}{6}\langle I'({\widetilde{u}}_0),{\widetilde{u}}_0\rangle \\&=\frac{1}{3}\Vert {\widetilde{u}}_0\Vert ^2+\frac{1}{12}L_{\phi _{{\widetilde{u}}_0}}({\widetilde{u}}_0) +\Bigl (\frac{1}{6}-\frac{1}{p+1}\Bigr )\int _{{\mathbb {R}}^3}|{\widetilde{u}}_0|^{p+1}dx\\&>\frac{1}{3}\Vert u_1\Vert ^2+\Bigl (\frac{1}{6}-\frac{1}{p+1}\Bigr )\int _{{\mathbb {R}}^3}|u_1|^{p+1}dx+\frac{1}{3}\Vert u_2\Vert ^2+\Bigl (\frac{1}{6}-\frac{1}{p+1}\Bigr )\int _{{\mathbb {R}}^3}|u_2|^{p+1}dx\\&\ \ \ +\frac{1}{12}L_{\phi _{u_1}}(u_1)+\frac{1}{12}L_{\phi _{u_2}}(u_2)+\frac{1}{6}L_{\phi _{u_1}}(u_2)\\&\ge \frac{s_v^2 }{3}\Vert u_1\Vert ^2+s_v^{p+1}\Bigl (\frac{1}{6}-\frac{1}{p+1}\Bigr )\int _{{\mathbb {R}}^3}|u_1|^{p+1}dx\\&\ \ \ +\frac{t_v^2 }{3}\Vert u_2\Vert ^2+t_v^{p+1}\Bigl (\frac{1}{6}-\frac{1}{p+1}\Bigr )\int _{{\mathbb {R}}^3}|u_2|^{p+1}dx\\&\ \ \ +\frac{s_v^4 }{12}L_{\phi _{u_1}}(u_1)+\frac{t_v^4 }{12}L_{\phi _{u_2}}(u_2)+\frac{s_v^2t_v^2}{6}L_{\phi _{u_1}}(u_2)\\&=\frac{1}{3}\Vert s_vu_1+t_vu_2\Vert ^2+\Bigl (\frac{1}{6}-\frac{1}{p+1}\Bigr )\int _{{\mathbb {R}}^3}|s_vu_1+t_vu_2|^{p+1}dx\\&\ \ \ +\frac{1}{12}L_{\phi _{s_vu_1+t_vu_2}}(s_vu_1+t_vu_2)\\&=I(s_vu_1+t_vu_2)-\frac{1}{6}\langle I'(s_vu_1+t_vu_2),s_vu_1+t_vu_2\rangle \\&=I(s_vu_1+t_vu_2)\ge m, \end{aligned}$$

which is a contradiction.

It remains to show that that $m>2c$. Indeed, in view of Lemma 2.4 (i), there exist ${\widetilde{s}},{\widetilde{t}}>0$ such that ${\widetilde{s}}{\widetilde{u}}^+_0,{\widetilde{t}}{\widetilde{u}}_0^-\in {\mathcal {N}}$. Thus, we infer from Lemma 2.2 that

$$\begin{aligned} \begin{aligned} m&=I({\widetilde{u}}_0)\ge I({\widetilde{s}}{\widetilde{u}}^+_0 +{\widetilde{t}}{\widetilde{u}}^-_0)\\&=I({\widetilde{s}}{\widetilde{u}}_0^+)+I({\widetilde{t}}{\widetilde{u}}_0^-)+\frac{{\widetilde{s}}^2{\widetilde{t}}^2}{2}L_{\phi _{{\widetilde{u}}_0^-}}({\widetilde{u}}_0^+)\\&>I({\widetilde{s}}{\widetilde{u}}_0^+)+I({\widetilde{t}}{\widetilde{u}}_0^-)\ge 2c. \end{aligned} \end{aligned}$$

$\square$