Stability of point defects of degree $\pm \frac{1}{2}$ in a two-dimensional nematic liquid crystal model

Abstract

We study k-radially symmetric solutions corresponding to topological defects of charge $\frac{k}{2}$ for integer $k\not = 0$ in the Landau-de Gennes model describing liquid crystals in two-dimensional domains. We show that the solutions whose radial profiles satisfy a natural sign invariance are stable when $|k| = 1$ (unlike the case $|k| > 1$ which we treated before). The proof crucially uses the monotonicity of the suitable components, obtained by making use of the cooperative character of the system. A uniqueness result for the radial profiles is also established.

1 Introduction

We consider the Landau-de Gennes model describing nematic liquid crystals through functions taking values into the space $\mathscr {S}_0$ of the so-called Q-tensors:

$$\begin{aligned} {\mathscr {S}}_0=\bigg \{Q\in {\mathbb {R}}^{3\times 3}\, :\, Q=Q^t, \mathrm{tr}(Q)=0\bigg \}, \end{aligned}$$

where $Q^t$ and $\mathrm{tr}(Q)$ are the transpose and the trace of Q.

We study critical points of the following Landau-de Gennes free energy functional:

$$\begin{aligned} {\mathscr {F}}(Q)= \int _{\Omega } \Big [\frac{1}{2}|\nabla {Q}|^2 + f_{bulk}(Q)\Big ]\,dx, \quad Q \in H^1_{loc}(\Omega , {\mathscr {S}}_0), \end{aligned}$$

(1.1)

where

$$\begin{aligned} f_{bulk}(Q) = -\frac{a^2}{2}\mathrm{tr}(Q^2)-\frac{b^2}{3}\mathrm{tr}(Q^3)+\frac{c^2}{4}\left( \mathrm{tr}(Q^2) \right) ^2, \end{aligned}$$

$a^2\ge 0$, $b^2, c^2>0$ and

$$\begin{aligned} \Omega =B_R\subset {\mathbb {R}}^2 \end{aligned}$$

is the disk centered at the origin of radius $R\in (0, \infty ]$ (if $R=\infty$, then $\Omega ={\mathbb {R}}^2$). These critical points satisfy the Euler–Lagrange system of equations:

$$\begin{aligned} \Delta Q=-a^2 Q-b^2\left[ Q^2-\frac{1}{3}|Q|^2I_3\right] +c^2|Q|^2\,Q \quad \text {in }\, \Omega , \end{aligned}$$

(1.2)

where $\frac{1}{3}|Q|^2=\frac{1}{3} \mathrm{tr}(Q^2)$ is the Lagrange multiplier associated to the tracelessness constraint. Recall that every critical point Q of ${\mathscr {F}}$ is smooth inside $\Omega$, see for instance [34].

The main goal of this article is to investigate the profile and energetic stability properties of certain symmetric solutions, the k-radially symmetric solutions, with $k=\pm 1$, that are physically relevant in describing the so-called “point defect” patterns.

Definition 1.1

Let $\Omega =B_R$ with $R\in (0, \infty ]$. For $k \in {\mathbb {Z}}{\setminus } \{ 0 \}$, we say that a Lebesgue measurable map $Q:\Omega \rightarrow {\mathscr {S}}_0$ is k-radially symmetric if the following conditions hold for almost every $x=(x_1, x_2)\in \Omega$:

• (H1) The vector $e_3 =(0,0,1)$ is an eigenvector of Q(x).

• (H2) The following identity holds

  $$\begin{aligned} Q\bigg (P_2\big ({\mathcal R}_2 (\psi ) {\tilde{x}}\big )\bigg )= {\mathcal R}_k (\psi ) Q(x) {\mathcal R}_k^t (\psi ) , \quad \text {for almost every } \psi \in {\mathbb {R}}, \end{aligned}$$

  where ${\tilde{x}}=(x_1,x_2,0)$, $P_2:{\mathbb {R}}^3\rightarrow {\mathbb {R}}^2$ is the projection given as $P_2(x_1,x_2,x_3)=(x_1,x_2)$ and

  $$\begin{aligned} {\mathcal R}_k (\psi ) := \left( \begin{array}{c@{\quad }c@{\quad }c} \cos \left( \frac{k}{2}\psi \right) &{} -\sin \left( \frac{k}{2}\psi \right) &{} 0 \\ \sin \left( \frac{k}{2}\psi \right) &{} \cos \left( \frac{k}{2}\psi \right) &{} 0 \\ 0 &{} 0 &{} 1 \end{array}\right) \end{aligned}$$

  (1.3)

  is the $\frac{k}{2}$-winding rotation around the vertical axis $e_3$.

Remark 1.2

In the previous work [26], we showed that if k is an odd integer, then a map $Q \in H^1 (\Omega , {\mathscr {S}}_0)$ satisfying (H2) automatically verifies (H1) (see Proposition 2.1 in [26]). Therefore for the case $|k|=1$ the hypothesis (H2) is sufficient.

We showed in [26] that when $\Omega$ is a ball $B_R$ of radius $R \in (0, \infty ]$ then k-radially symmetric solutions of (1.2) have a simple structure:

$$\begin{aligned} Q(x)= u(|x|) \sqrt{2}\left( n(x)\otimes n(x)-\frac{1}{2}I_2\right) + v(|x|) \sqrt{\frac{3}{2}}\left( e_3\otimes e_3-\frac{1}{3}I_3\right) , \end{aligned}$$

(1.4)

where the vector field n is given by

$$\begin{aligned} n(r\cos \varphi , r\sin \varphi ) =\left( \cos \left( {\textstyle \frac{k}{2}} \varphi \right) , \sin \left( {\textstyle \frac{k}{2}} \varphi \right) , 0\right) , \quad r>0, \varphi \in [0, 2\pi ), \end{aligned}$$

(1.5)

$I_3$ is the $3\times 3$ identity matrix, $I_2=I_3-e_3\otimes e_3$, and (u, v) satisfies on (0, R) the following system of ODEs:

$$\begin{aligned} {\left\{ \begin{array}{ll} u''+\frac{u'}{r}-\frac{k^2u}{r^2} = h(u,v) \\ v''+\frac{v'}{r}= g(u,v), \end{array}\right. } \end{aligned}$$

(1.6)

with

$$\begin{aligned} h(u,v)&= u\Big [-a^2+\sqrt{\frac{2}{3}} b^2 v+c^2\left( u^2+ v^2\right) \Big ], \end{aligned}$$

(1.7)

$$\begin{aligned} g(u,v)&= v\Big [-a^2-\frac{1}{\sqrt{6}}b^2 v+c^2\left( u^2+ v^2\right) \Big ] + \frac{1}{\sqrt{6} } b^2 u^2. \end{aligned}$$

(1.8)

We couple the Eq. (1.2) with the boundary conditions that are physically motivated and compatible with the k-radial symmetry:

$$\begin{aligned} Q(x) = Q_k(x):=s_+ \left( n (x) \otimes n(x) -\frac{1}{3} I_3 \right) \quad \hbox { as } x \in \partial B_R, \end{aligned}$$

(1.9)

where the map $n:\partial B_r\rightarrow {\mathbb {S}}^2$ is given by (1.5) and

$$\begin{aligned} s_+=\frac{ b^2 + \sqrt{b^4+24 a^2 c^2}}{4 c^2}>0. \end{aligned}$$

(1.10)

When $R = \infty$, Eq. (1.9) should be understood as

$$\begin{aligned} \lim _{|x| \rightarrow \infty } |Q(x) - Q_k(x)| = 0. \end{aligned}$$

These boundary conditions also carry a topological information by having in a suitable sense a “$\frac{k}{2}$ degree” for n, see the next subsection for details. Moreover, the boundary condition (1.9), together with the singular character of the ODE at the origin lead to the following boundary conditions for the ODE system:

$$\begin{aligned} u(0)=0,{\ v'(0)=0}, \ u(R)=\frac{1}{\sqrt{2}} s_+,\quad v(R)=-\frac{1}{\sqrt{6}}s_+. \end{aligned}$$

(1.11)

(When $R=\infty$ we naturally define the boundary conditions in the limiting sense.)

The physical reasons for the study of these solutions are given in the next subsection, that the mathematically-oriented reader may safely skip to reach the subsection detailing the main results.

1.1 Physical background

The Q-tensors describe the main characteristic feature of the nematic liquid crystal material, namely the local orientational ordering of the rod-like molecules and can be regarded as a crude measure of the local alignment (see [14, 36] for details).

The simplest predictions are obtained by using Q-tensor valued maps in a free energy, whose minimizers describe equilibrium states. The type of free energy that we consider here is the simplest one that still captures fundamental physical aspects. The gradient part of the free energy density of a Q-tensor map, namely $|\nabla Q|^2=\sum \nolimits _{k=1}^2\sum \nolimits _{i,j=1}^3 \big (\frac{\partial Q_{ij}}{\partial x_k}\big )^2$ penalises the spatial variations while the bulk potential $f_{bulk}(Q)$ captures the specific liquid crystal aspects. It can be regarded as a Taylor-expansion (around the isotropic state $Q=0$) that respects the physical invariance $f_{bulk}(Q)=f_{bulk}(RQR^t)$ for $R\in SO(3)$ (see [14, 36] for details). The regime we consider (by choosing the sign of the coefficient in front of $\mathrm{tr}(Q)^2$ in $f_{bulk}$) is the deep nematic regime, in which case the isotropic state $Q=0$ is an unstable critical point of the bulk potential. In general, thanks to suitable scalings [18, 26] one can physically think of the regime when $a^2,c^2$ fixed and $b^2\rightarrow 0$ as the “low temperature regime”, and we will use this terminology throughout the paper.

Set

$$\begin{aligned} s_-=\frac{b^2 - \sqrt{b^4+24a^2c^2}}{4c^2} < 0. \end{aligned}$$

(1.12)

The bulk potential has two sets of local minima, namely,

$$\begin{aligned} \left\{ s_-\left( n\otimes n-\frac{1}{3}I_3\right) , n\in \mathbb {S}^2\right\} \text { and } \left\{ s_+\left( n\otimes n-\frac{1}{3}I_3\right) , n\in \mathbb {S}^2\right\} , \end{aligned}$$

where the former set contains local minimizers while the latter one contains all the global minimizers.

We choose the boundary conditions that are k-radially symmetric and belong to the set of global minimizers, as this allows for a direct comparison with the simpler director or Oseen-Frank theory and most importantly leads to a study of liquid crystal defect profiles. Furthermore, one notes that the set

$$\begin{aligned} {\mathscr {S}}_*=\left\{ s_+\left( n\otimes n-\frac{1}{3}I_3\right) \, :\, n\in \mathbb {S}^2\right\} \end{aligned}$$

(1.13)

is homeomorphic to ${\mathbb {R}}P^2$ while the smaller set in which we consider boundary conditions, namely

$$\begin{aligned} {\mathscr {S}}_*^{lim}=\left\{ s_+\left( n\otimes n-\frac{1}{3}I_3\right) , n\in \mathbb {S}^1\right\} \end{aligned}$$

(1.14)

is homeomorphic to $\mathbb {R}P^1$. Moreover, if we consider $Q_k$ given by (1.9) as an $\mathbb {R}P^1$-valued map on ${\mathbb {R}}^2 {\setminus } \{0\}$, then it has degree k / 2 about the origin. (For a definition of the degree for $\mathbb {R}P^1$-valued maps, see for instance [8, pp. 685–686]).

This model can be seen as the 2D reduction of the physical situation of a 3D cylindrical boundary domain, with so-called “homeotropic” lateral boundary conditions where the configurations are invariant in the vertical direction (see for instance [6]). Its main validation at a physical level is related to its capacity of describing certain patterns which provide the most striking optical signature of the liquid crystal and the very reason for the “nematic” name (with nematic being related to a Greek word meaning “thread”). These patterns are referred to as “defect” patterns and are characterised by significant and highly localised variations in the material properties. There are several types of defect patterns, the point defects being the simplest (see [10, 29, 30]). Nevertheless, despite their apparent simplicity the analytical investigation of their structure and profile generates very challenging nonlinear analysis problems [3–5, 9, 12, 15, 17, 22, 24–27, 33].

One can classify topologically the 2D point defects, by the topological degree of the so-called “optical eigenvector” namely the eigenvector corresponding to the largest eigenvalue (assuming that this is also restricted to a plane). Thus the symmetric solutions we described are the prototypical types of defects, the most symmetrical such types of defects.

There is a direct analogy with the Ginzburg–Landau theory of superconductivity, where the defects are also classified topologically and it is known that only the “lowest degree” defects are stable, but not the higher degree ones, see [35]. The Ginzburg–Landau system exhibits a number of analogies with our case, however there are significant differences and additional difficulties in our case, see for instance the discussion in [24, 27]. We would like to remark that this analogy can be quite misleading in certain circumstances, for example, in the context of 3D Landau-de Gennes theory, the melting hedgehog which is in a (debatable) sense the “lowest degree” defect, can be unstable in a certain temperature regime [18, 24].

1.2 Main mathematical results

In [26], we constructed solutions (u, v) of (1.6) and (1.11) in (0, R) with $R\in (0, \infty ]$ using variational methods. These solutions give rise to k-radially symmetric solutions Q of the Euler–Lagrange equations (1.2) with the boundary conditions (1.9) via (1.4). Furthermore, these solutions satisfy

$$\begin{aligned} u>0 \quad \text { and }\quad v<0 ~\text { in }(0,R) \end{aligned}$$

and they are local minimizers of the corresponding energy functional, in the sense that, for any $R' < R$ and any $(\xi , \eta ) \in C_c^\infty (0,R')$ satisfying

$$\begin{aligned} \sup _{(0,R')}|\eta | < \min \left( \frac{s_+}{ \sqrt{6}}, \sqrt{\frac{2}{3}}|s_-|\right) \end{aligned}$$

there holds

$$\begin{aligned} {\mathscr {E}}_{R'}(u,v) \le {\mathscr {E}}_{R'}(u + \xi , v + \eta ), \end{aligned}$$

where

$$\begin{aligned} {\mathscr {E}}_{R'}(u,v)&= \int _0^{R'} \bigg [ \frac{1}{2} \left( (u')^2+(v')^2+\frac{k^2}{r^2}u^2 \right) + f(u,v)\bigg ]\,rdr,\end{aligned}$$

(1.15)

$$\begin{aligned} f(u,v)&=-\frac{a^2}{2}(u^2 + v^2)+\frac{c^2}{4}\left( u^2+v^2\right) ^2 -\frac{b^2}{3 \sqrt{6}}v(v^2 -3u^2). \end{aligned}$$

(1.16)

(Note that

$$\begin{aligned} h(p,q)= \frac{\partial f}{\partial p} (p,q) \text { and }g(p,q)=\frac{\partial f}{\partial q}(p,q), \end{aligned}$$

and so (1.6) is the Euler–Lagrange equation for ${\mathscr {E}}_{R'}$.)

When $R < \infty$, we can allow $R' = R$ in the above definition of local minimality. However, on the infinite domain $(0, \infty )$, the energy ${\mathscr {E}}_\infty (u,v)=\infty$, and therefore the local minimality property of (u, v) should be understood as above with any $R' < R=\infty$.

Our main result is the stability of the critical points Q on $B_R$ for $R\in (0, \infty ]$, defined by (1.4) corresponding to any stable solutions (u, v) as above, of the Landau-de Gennes energy ${\mathscr {F}}$ in the case $k = \pm 1$. For any solution Q of (1.2) subjected to the boundary condition (1.9) we define the second variation ${\mathscr {L}}[Q] (P)$ at Q in direction $P \in C_c^\infty (B_{R'}, {\mathscr {S}}_0)$ ($R' < R$) as follows:

$$\begin{aligned} {{\mathscr {L}}}[Q](P)= & {} \frac{d^2}{dt^2}\Big |_{t=0} \int _{B_{R'}} \Big \{\frac{1}{2}|\nabla (Q+tP)|^2 + f_{bulk}(Q + tP)\Big \}\,dx\nonumber \\= & {} \int _{B_R}\Big \{|\nabla P|^2-{a^2}|P|^2-2b^2\mathrm{tr}(P^2 Q)+{c^2}\left( |Q|^2|P|^2+2|\mathrm{tr}(QP)|^2\right) \Big \}\,dx.\nonumber \\ \end{aligned}$$

(1.17)

This definition extends to $P \in H_0^1(B_R, {\mathscr {S}}_0)$ for $R\in (0, \infty ]$ (recall that $H_0^1({\mathbb {R}}^2, {\mathscr {S}}_0)=H^1({\mathbb {R}}^2, {\mathscr {S}}_0)$).

A related issue is the stability of the ODE solution (u, v) on (0, R) for $R\in (0, \infty ]$. The second variation for ${\mathscr {E}}_R$ at a solution (u, v) of (1.6) and (1.11) in direction $(\xi ,\eta ) \in C_c^\infty (0,R')$, ($R'<R$), is defined similarly as

$$\begin{aligned} B(\xi ,\eta )= & {} \frac{d^2}{dt^2}\Big |_{t = 0} {\mathscr {E}}_{R'}(u + t\xi , v + t\eta )\nonumber \\= & {} \int _{0}^R \bigg \{ | \xi '|^2+\frac{k^2}{r^2} |\xi |^2 + | \eta ' |^2 + \left( -{a^2} + \frac{2b^2}{\sqrt{6}} v + c^2 (3u^2+v^2) \right) |\xi |^2 \nonumber \\&+ \left( -{a^2} - \frac{2b^2}{\sqrt{6}} v + c^2 (u^2+3 v^2) \right) |\eta |^2 + 4 u \xi \eta \left( \frac{b^2}{\sqrt{6}} + c^2 v \right) \bigg \}\, rdr.\qquad \end{aligned}$$

(1.18)

This definition extends to $(\xi ,\eta ) \in {\hat{X}}_R$, where ${\hat{X}}_R$ is the completion of $C_c^\infty (0,R)$ under the norm

$$\begin{aligned} \Vert (\xi ,\eta )\Vert _{{\hat{X}}_R}^2 = \int _0^R \Big [|\xi '|^2 + |\eta '|^2 + (1 + r^{-2})|\xi |^2 + |\eta |^2\Big ]\,r\,dr. \end{aligned}$$

In fact,

$$\begin{aligned}&\bullet \text { if } R \in (0,\infty ), \quad {\hat{X}}_R = \Big \{(\xi ,\eta ): [0,R] \rightarrow {\mathbb {R}}^2\,\Big | \sqrt{r}\xi ', \sqrt{r}\,\eta ', \frac{1}{\sqrt{r}}\xi , \sqrt{r}\eta \in L^2(0,R),\\&\qquad \qquad \qquad \qquad \qquad \xi (R) = \eta (R) = 0\Big \},\\&\bullet \text { if } R=\infty , \quad {\hat{X}}_\infty = \left\{ (\xi ,\eta ): [0,\infty ) \rightarrow {\mathbb {R}}^2\,\Big | \sqrt{r}\xi ', \sqrt{r}\,\eta ', \left( \frac{1}{\sqrt{r}} + \sqrt{r}\right) \xi , \sqrt{r}\eta \in L^2(0,\infty )\right\} . \end{aligned}$$

We refer to Lemma 3.1 below for the behavior of $(\xi ,\eta ) \in {\hat{X}}_R$.

We recall our previous result from [26] on the instability of k-radially symmetric solutions Q in ${\mathbb {R}}^2$ for $|k|>1$:

Theorem 1.3

([26]) Assume that¹ $a^2 \ge 0$, $b^2, c^2 > 0$ and $|k| > 1$. Let (u, v) be any solution of (1.6) on $(0,\infty )$ under the boundary condition (1.11) (with $R = \infty$) such that $u > 0$ and $v < 0$ on $(0,\infty )$. Then the solution Q of (1.2) on ${\mathbb {R}}^2$ given by (1.4) and satisfying the boundary condition (1.9) is unstable with respect to ${\mathscr {F}}$, namely there exists $P \in C_c^\infty ({\mathbb {R}}^2,{\mathscr {S}}_0)$ such that ${\mathscr {L}}[Q](P) < 0$.

We complete the study of k-radially symmetric critical points of ${\mathscr {F}}$ with the following stability result for $k = \pm 1$ in any disk $B_R$ with $R \in (0, \infty ]$.

Theorem 1.4

(Stability) Assume that $a^2 \ge 0$, $b^2, c^2 > 0$ and $k = \pm 1$. Let $R \in (0, \infty ]$ and (u, v) be any solution of (1.6) on (0, R) under the boundary condition (1.11) such that $u > 0$ and $v < 0$. Assume further that (u, v) is stable with respect to ${\mathscr {E}}_R$, i.e.

$$\begin{aligned} B(\xi ,\eta ) \ge 0 \quad \text { for all } (\xi ,\eta ) \in {\hat{X}}_R. \end{aligned}$$

(1.19)

Then the solution Q of (1.2) on $B_R$ given by (1.4) and satisfying the boundary condition (1.9) is stable with respect to ${\mathscr {F}}$, i.e. ${\mathscr {L}}[Q] (P) \ge 0$ for all $P \in H_0^1(B_R, {\mathscr {S}}_0)$.

Furthermore, ${\mathscr {L}}[Q](P) = 0$ for some $P \in H^1_0(B_R,{\mathscr {S}}_0)$ if and only if, for some $(\xi _0,\eta _0) \in {\hat{X}}_R$ satisfying $B(\xi _0,\eta _0) = 0$ and some constants $\alpha$, $\beta$,

• either $R = \infty$ and

  $$\begin{aligned} P(x)= & {} \xi _0(|x|) \sqrt{2}\left[ n(x)\otimes n(x)-\frac{1}{2}I_2\right] + \eta _0(|x|) \sqrt{\frac{3}{2}}\left[ e_3\otimes e_3-\frac{1}{3}I_3\right] \\&+\, \alpha \frac{\partial Q}{\partial x_1}(x) + \beta \frac{\partial Q}{\partial x_2}(x), \end{aligned}$$

• or $R < \infty$ and

  $$\begin{aligned} P(x) = \xi _0(|x|) \sqrt{2}\left[ n(x)\otimes n(x)-\frac{1}{2}I_2\right] + \eta _0(|x|) \sqrt{\frac{3}{2}}\left[ e_3\otimes e_3-\frac{1}{3}I_3\right] , \end{aligned}$$

where n(x) is given by (1.5).

Remark 1.5

Loosely speaking, the second part of Theorem 1.4 asserts that the kernel of ${\mathscr {L}}[Q]$ is generated by the kernel of the second variation B of ${\mathscr {E}}_R$ at (u, v) and $span\{\partial _{x_1} Q, \partial _{x_2} Q\}$.

Two-dimensional point defects in the Landau-de Gennes framework have been studied for quite some time in the literature; see e.g. [5, 9, 11, 15, 17, 21, 23, 26, 31, 32] (and also [16, 28] in micromagnetics). Our motivation came from the paper [15] which concerns the extreme low-temperature regime ($b^2=0$). It was shown therein that there exists a unique global minimizer of the Landau-de Gennes energy which is k-radially symmetric and provides the description of the ground state profile of a point defect of index k / 2. We followed this up in [26] with the case $b^2 > 0$ and established the instability of entire k-radially symmetric solutions when $|k| > 1$.

Different but related questions were considered on more general domains and for more general boundary conditions in [5, 9, 21, 23]. To put Theorem 1.4 in perspective, we draw attention to [5, 9, 21]. In [5], the Landau-de Gennes energy was investigated for functions taking values into a restricted three dimensional space of Q-tensors. It was shown that, in the case of small elastic constant, the minimizers of Landau-de Gennes energy exhibit behavior similar to those of Ginzburg–Landau energy [7], namely for boundary conditions of degree k / 2 there are exactly k vortices of degree $\pm 1/2$. In [9, 21] the minimizers of the full Landau-de Gennes energy were studied under non-orientable boundary conditions (which in our setting amounts to k being odd). It was shown that in the low temperature regime and in the case of small elastic constant the minimizer has only one vortex.

The proof of Theorem 1.4 uses the type of framework we set up to treat the analogous problem of stability/instability of the melting hedgehog in three dimensions [24, 25, 27]. The first step of the proof entails a careful choice of basis decomposition for ${\mathscr {S}}_0$ so that the problem reduces, via Fourier decompositions, to an infinite set of partially coupled problems which involve functions of only one variable. In a loose sense, this can be viewed as some kind of partial separation of variables. The reduced problem for each Fourier mode is then treated using the so-called Hardy decomposition tricks together with certain qualitative properties of the profile functions u and v. In particular, the following monotonicity result is of special importance in our proof.

Theorem 1.6

(Monotonicity) Assume that $a^2 \ge 0$, $b^2, c^2 > 0$ and $k \ne 0$. Let $R \in (0, \infty ]$ and (u, v) be any solution of (1.6) on (0, R) under the boundary condition (1.11) such that $u > 0$ and $v < 0$.

• If $b^4 < 3a^2c^2$, then $u'>0$ and $v'>0$ in (0, R).

• If $b^4 = 3a^2c^2$, $u' > 0$ in (0, R) while $v \equiv -\frac{s_+}{\sqrt{6}}$.

• If $b^4 > 3a^2c^2$, then $u'>0$ while $v'<0$ in (0, R).

Regarding the assumption that (u, v) is stable for ${\mathscr {E}}_R$ in Theorem 1.4, we note that the solution (u, v) constructed in [26] (for any given $a^2 \ge 0$, $b^2, c^2 > 0$) is a local minimizer and thus stable. In fact, for “small b”, we have the following uniqueness and strict stability result.

Theorem 1.7

(Uniqueness) Assume that $a^2, b^2, c^2 > 0$, $R \in (0, \infty ]$ and $k \ne 0$. If $b^4 \le 3a^2c^2$, under the assumption that $u > 0$ and $v < 0$, there exists a unique solution of (1.6) on (0, R) under the boundary condition (1.11). Furthermore such (u, v) is strictly stable in the sense that $B(\xi ,\eta ) > 0$ for all $(\xi ,\eta ) \in {\hat{X}}_R$, $(\xi ,\eta ) \not \equiv 0$. In particular, (u, v) is a local minimizer of ${\mathscr {E}}_R$.

The results above lead to the following open problem.

Open problem 1.8

Are solutions of (1.6) and (1.11) (with or without the assumption that $u > 0$, $v < 0)$ unique?

Remark 1.9

The case $b^4=3a^2c^2$ was proved earlier in [26] using a different method. A careful mixture of the proof of Theorem 1.7 in Sect. 3 below and various estimates in [26] shows that Theorem 1.7 continues to hold for $b^4 \le \frac{75}{7} a^2\,c^2$. However, since this can be shown to be non-sharp and there is a distinctive difference between the case $b^4 > 3a^2c^2$ and the case $b^4 \le 3a^2c^2$ (e.g. change of the monotone behaviour of v), we have chosen to keep the statement of the result as above. It remains an open question if uniqueness holds for all a, b and c.

The rest of paper is structured as follows. In Sect. 2, we prove the monotonicity of u and v assuming sign constraints $u>0$ and $v<0$. In Sect . 3, we prove fine properties of functions $(\xi , \eta )\in {\hat{X}}_R$ (see Lemma 3.1) and the uniqueness result of Theorem 1.7. Section 4 is devoted to the proof of the stability result in Theorem 1.4. Finally, in the appendix, we include a calculus lemma which is needed in the body of the paper.

2 Monotonicity

In this section we prove monotonicity of solutions (u, v) of the system (1.6). Let us fix some $R \in (0,+\infty ]$ and consider the ODE system (1.6) on (0, R) subjected to the boundary condition (1.11).

Assume further that²

$$\begin{aligned} u > 0\quad \text { and }\quad v < 0 \text { in } (0,R). \end{aligned}$$

We showed in [26, Propositions 3.4, 3.5, 3.7] that

$$\begin{aligned} 0< u < \frac{s_+}{\sqrt{2}} \text { and } \min \left( - \frac{s_+}{\sqrt{6}}, \frac{2s_-}{\sqrt{6}}\right) \le v \le \max \left( - \frac{s_+}{\sqrt{6}}, \frac{2s_-}{\sqrt{6}}\right) \quad \text { in } (0,R), \end{aligned}$$

(2.1)

with equality holding only if $b^4=3a^2c^2$, in which case $v\equiv -\frac{s_+}{\sqrt{6}}=\frac{2s_-}{\sqrt{6}}$.

Furthermore (see Step 3 of the proof of Proposition 3.1 in [26])

$$\begin{aligned} \sqrt{3} v + u < 0 \text { in }(0,R). \end{aligned}$$

(2.2)

The monotonicity of u and v depends on how big $b^4$ is relative to $a^2c^2$. When $b^4 = 3a^2c^2$, one has $v \equiv -\frac{s_+}{\sqrt{6}}$ and u is the unique solution of the ODE

$$\begin{aligned} u'' + \frac{u'}{r} - \frac{k^2}{r^2}u = c^2\,u\left( u^2 - \frac{s_+^2}{2}\right) , u(0) = 0, s(R) = \frac{s_+}{\sqrt{2}}. \end{aligned}$$

For other values of b, the monotonicity of u and v will be proved using the theory of cooperative systems and the moving plane method (see e.g. [13]).

Proof of Theorem 1.6

When $b^4 = 3a^2c^2$ the result is a consequence of [26, Proposition 3.5] and [25, Lemma 3.7]. We assume for the rest of the proof that $b^4 \ne 3a^2c^2$.

Case 1: $b^4 < 3a^2c^2$.

First assume that $R < \infty$. We note that (see [26])

$$\begin{aligned} \partial _q h(p,q)= & {} \partial _p g(p,q)< 0\quad \text { for all }0< p< \frac{s_+}{\sqrt{2}} \text { and } \min \left( - \frac{s_+}{\sqrt{6}}, \frac{2s_-}{\sqrt{6}}\right) \\<&q < \max \left( - \frac{s_+}{\sqrt{6}}, \frac{2s_-}{\sqrt{6}}\right) . \end{aligned}$$

This plays an important role in our argument below.

For $0< s < R$, define

$$\begin{aligned} u_s(r) = u(2s - r) \text { and } v_s(r) = v(2s - r) \text { for } \max (0,2s - R)< r < s. \end{aligned}$$

Note that $h(u(R),v(R)) = g(u(R),v(R)) = 0$ and recall that $0< u < u(R)$ and $v < v(R)$ in (0, R) [thanks to (2.1)]. In particular, the function ${\hat{u}} = u - u(R)$ satisfies

$$\begin{aligned} {\hat{u}}'' + \frac{1}{r} {\hat{u}}' - \frac{k^2}{r^2}\,{\hat{u}} = \frac{k^2}{r^2} u(R) + h(u,v) - h(u(R),v(R)) \ge h(u,v) - h(u(R),v) = \xi \,{\hat{u}} \end{aligned}$$

for some function $\xi \in C[0,R]$. As ${\hat{u}}(R) = 0$ and ${\hat{u}} < 0$ in (0, R), we deduce from the Hopf lemma (see e.g. [20, Lemma 3.4]) that $u'(R) > 0$. Likewise, we can show that $v'(R) > 0$. Consequently, there is some small $\epsilon > 0$ such that $u_s > u$ and $v_s > v$ in $\max (0,2s - R)< r < s$ for any $R - \epsilon< s < R$.

We define

$$\begin{aligned} \underline{s} \!=\! \inf \Big \{0 \!<\! s \!<\! R : u_{s'}> u \text { and }v_{s'} > v \text { in }\max (0,2{s'} - R)< r < s' \quad \text { for all } s' \in (s,R)\Big \}, \end{aligned}$$

then $\underline{s} \in [0,R)$.

We claim that $\underline{s}=0$. Assume by contradiction that $\underline{s}>0$, then,

• (i) $u' \ge 0$ and $v' \ge 0$ in $(\underline{s},R)$,

• (ii) and $u_{\underline{s}} \ge u > 0$ and $v_{\underline{s}} \ge v$ in $\max (0,2{\underline{s}} - R)< r < \underline{s}$.

It follows that

$$\begin{aligned} u_{\underline{s}}'' + \frac{1}{r}u_{\underline{s}}' - \frac{k^2}{r^2}\,u_{\underline{s}}&\le h(u_{\underline{s}},v_{\underline{s}}) \le h(u_{\underline{s}},v),\end{aligned}$$

(2.3)

$$\begin{aligned} v_{\underline{s}}'' + \frac{1}{r}v_{\underline{s}}'&\le g(u_{\underline{s}},v_{\underline{s}}) \le g(u,v_{\underline{s}}) \text { in } \max (0,2{\underline{s}} - R)< r < \underline{s}, \end{aligned}$$

(2.4)

and so

$$\begin{aligned} (u_{\underline{s}}-u)'' + \frac{1}{r}(u_{\underline{s}}-u)' - \frac{k^2}{r^2}\,(u_{\underline{s}}-u)&\le h(u_{\underline{s}},v) - h(u,v)=(u_{\underline{s}}-u)c_1(r),\end{aligned}$$

(2.5)

$$\begin{aligned} (v_{\underline{s}}-v)'' + \frac{1}{r}(v_{\underline{s}}-v)'&\le g(u,v_{\underline{s}}) - g(u,v)=(v_{\underline{s}}-v)c_2(r), \end{aligned}$$

(2.6)

with $c_1, c_2$ being two continuous functions on $[\max (0,2{\underline{s}} - R), \underline{s}]$.

Noting that, by (1.11) and (2.1),

$$\begin{aligned} u_{\underline{s}}(\max (0,2{\underline{s}} - R))&> u(\max (0,2{\underline{s}} - R)),\\ u_{\underline{s}}(\underline{s})&= u(\underline{s}), \end{aligned}$$

we can appeal to the strong maximum principle and the Hopf lemma to conclude that

$$\begin{aligned} u_{\underline{s}}> u \text { in }\max (0,2{\underline{s}} - R)< r < \underline{s} \text { and } u_{\underline{s}}'(\underline{s}) > u'(\underline{s}). \end{aligned}$$

(2.7)

This implies that the second inequality in (2.4) is strict and so is the inequality in (2.6). We again apply the strong maximum principle and the Hopf lemma to obtain

$$\begin{aligned} v_{\underline{s}}> v \text { in }\max (0,2{\underline{s}} - R)< r < \underline{s} \text { and } v_{\underline{s}}'(\underline{s}) > v'(\underline{s}). \end{aligned}$$

(2.8)

Estimates (2.7) and (2.8) contradict the minimality of $\underline{s}$. Therefore, $\underline{s} = 0$ as claimed. This proves that $u', v'\ge 0$ on (0, R).

We turn to show that $u', v'> 0$ on (0, R). We have the following equations for $(u', v')$:

$$\begin{aligned} u'''+\frac{u''}{r}-\frac{(k^2+1)u'}{r^2} + \frac{2 k^2u}{r^3}&={u'} \partial _u h(u,v) + v'\,\partial _v h(u,v),\nonumber \\ v'''+\frac{v''}{r}-\frac{v'}{r^2}&={v'} \partial _v g(u,v) + u'\,\partial _u g(u,v). \end{aligned}$$

(2.9)

Noting now that $\partial _v h(u,v) = \partial _u g(u,v) < 0$, we arrive at

$$\begin{aligned} u'''+\frac{u''}{r}-\frac{(k^2+1)u'}{r^2}&\le {u'} \partial _u h(u,v) ,\\ v'''+\frac{v''}{r}-\frac{v'}{r^2}&\le {v'} \partial _v g(u,v) . \end{aligned}$$

Since $u'(R) > 0$ and $v'(R) > 0$, the strong maximum principle implies that $u'>0$ and $v'>0$, as desired.

Next, consider the case $R = \infty$. In order for the above argument to carry through, we need to show that there is some $R_0 > 0$ such that

$$\begin{aligned} u'> 0\quad \text { and }\quad v' > 0 \text { in } (R_0,\infty ). \end{aligned}$$

To this end, recall the asymptotics (see [26])

$$\begin{aligned} u (r)&= \frac{s_+}{\sqrt{2}} + p_1\,r^{-2} + O(r^{-4}),\end{aligned}$$

(2.10)

$$\begin{aligned} v(r)&= -\frac{s_+}{\sqrt{6}} + q_1\,r^{-2} + O(r^{-4}), \end{aligned}$$

(2.11)

where $p_1 = - \frac{\sqrt{2}k^2}{2}\,\frac{2b^2 + c^2\,s_+}{b^2(-b^2 + 4c^2\,s_+)}$, $q_1 = - \frac{\sqrt{6}k^2}{2}\,\frac{-b^2 + c^2\,s_+}{b^2(-b^2 + 4c^2\,s_+)}$, and the big ‘O’ notation is meant for large r. In fact, the argument therein leads to an asymptotic expansion

$$\begin{aligned} u (r)&= \frac{s_+}{\sqrt{2}} + p_1\,r^{-2} + p_2\,r^{-4} + \cdots + p_N\,r^{-2N} + O(r^{-2N -2}),\\ v(r)&= -\frac{s_+}{\sqrt{6}} + q_1\,r^{-2} + q_2\,r^{-4} + \cdots + q_N\,r^{-2N} + O(r^{-2N -2}), \end{aligned}$$

for any given $N \ge 2$ and with explicitly computable coefficients $p_i$, $q_i$’s. In particular, we obtain that

$$\begin{aligned} \frac{1}{r}(ru')'&= u'' + \frac{u'}{r} = h(u,v) + \frac{k^2}{r^2}u = 4p_1\,r^{-4} + O(r^{-6}),\end{aligned}$$

(2.12)

$$\begin{aligned} \frac{1}{r}(rv')'&=v'' + \frac{v'}{r} = g(u,v) = 4q_1\,r^{-4} + O(r^{-6}). \end{aligned}$$

(2.13)

For $b^4 < 3a^2c^2$, we have $p_1 < 0$ and $q_1 < 0$. Hence, there is some $R_0 > 0$ such that $(ru')' < 0$ and $(rv')' < 0$ in $(R_0,\infty )$. Integrating twice, it follows that for any $R_0<s<r$, we have

$$\begin{aligned} u(s)+su'(s)\log \frac{r}{s}\ge u(r), \quad v(s)+sv'(s)\log \frac{r}{s}\ge v(r). \end{aligned}$$

As u and v have a limit as $r\rightarrow \infty$, this implies that $u' \ge 0$ and $v' \ge 0$ in $(R_0,\infty )$. Since $(ru')' < 0$ and $(rv')' < 0$ in $(R_0,\infty )$ we conclude that $u'>0$ and $v'>0$ in $(R_0, \infty )$. This completes the proof when $b^4 < 3a^2c^2$.

Case 2: $b^4 > 3a^2c^2$. The proof is similar except the following changes:

• $\partial _v h(u,v) = \partial _u g(u,v) > 0$ for all (u, v) satisfying (2.1),

• $v'(R) < 0$ if $R < \infty$,

• $q_1 > 0$ if $R = \infty$.

We omit the details. $\square$

3 Uniqueness

In this section, we prove Theorem 1.7.

Strategy. In order to prove the uniqueness of the solution of the ODE system (1.6), (1.11), for which $u>0$ and $v<0$, we follow a strategy similar to that in [2]. We show that any solution of the ODE system with the mentioned signs of the components is a local minimizer if $b^4\le 3a^2c^2$ (see Proposition 3.8). Then the uniqueness will follow by the mountain pass lemma. Indeed, assuming by contradiction that there exist two such solutions, we use a mountain pass argument to find another solution on a trajectory connecting the two given ones. This solution will have to be unstable thus leading to a contradiction and proving the uniqueness. Many of the complications in our treatment are because of the loss of compactness due to the infinite domain (i.e. when $R = \infty$).

Some notation. Recall

$$\begin{aligned} {\mathscr {E}}_R(u,v)&= \int _0^R \bigg [ \frac{1}{2} \left( (u')^2+(v')^2+\frac{k^2}{r^2}u^2 \right) + f(u,v)\bigg ]\,rdr, \end{aligned}$$

where

$$\begin{aligned} f(u,v) = -\frac{a^2}{2}(u^2 + v^2)+\frac{c^2}{4}(u^2+v^2)^2 -\frac{b^2}{3 \sqrt{6}}v(v^2 -3u^2) \end{aligned}$$

and

$$\begin{aligned} h(u,v)=\partial _u f(u,v), \quad g(u,v)=\partial _v f(u,v). \end{aligned}$$

For $0< R < \infty$, if we define

$$\begin{aligned} X_R = \Big \{(u,v): [0,R] \rightarrow {\mathbb {R}}^2\,\Big | \sqrt{r}u', \sqrt{r}\,v', \frac{1}{\sqrt{r}}u, \sqrt{r}v \in L^2(0,R), u(R) = \frac{s_+}{\sqrt{2}}, v(R) = -\frac{s_+}{\sqrt{6}}\Big \}, \end{aligned}$$

then ${\mathscr {E}}_R \in C^1(X_R,{\mathbb {R}})$.

For $R = \infty$, we have a complication as $\frac{u^2}{r}$ and $r\,f(u,v)$ are not integrable over $(0,\infty )$. To fix this issue, it is useful to note that if $(u_1, v_1)$ and $(u_2,v_2)$ are two solutions of (1.6) and (1.11), then, thanks to the asymptotic estimate (2.10),

$$\begin{aligned} \Big (\frac{1}{\sqrt{r}} + \sqrt{r}\Big )\,(u_1 - u_2) \in L^2(0,\infty ). \end{aligned}$$

To accommodate both situations, we let $(u_0,v_0)$ be a fixed solution of (1.6) and (1.11), satisfying $u_0 \ge 0$, $v_0 \le 0$ (then by [26] $u_0>0$, $v_0<0$ on (0, R)). Consider instead of ${\mathscr {E}}_R$ the modified functional for $R\in (0, \infty ]$:

$$\begin{aligned} {\hat{{\mathscr {E}}}}_R({\hat{u}}, {\hat{v}})= & {} \int _0^R \frac{1}{2} \Big ( |(u_0 + {\hat{u}})'|^2 - |u_0'|^2 +|(v_0 +{\hat{v}})'|^2 - |v_0'|^2\Big )\,r\,dr\\&+ \int _0^R \frac{k^2}{2r}((u_0 + {\hat{u}})^2 - u_0^2)\,dr \\&+ \int _0^R \big [f(u_0 + {\hat{u}},v_0 + {\hat{v}}) - f(u_0,v_0)\big ]\,rdr, \end{aligned}$$

where $({\hat{u}}, {\hat{v}})$ belongs to ${\hat{X}}_R$ defined by

$$\begin{aligned}&\bullet \text { if } R \in (0,\infty ), \quad {\hat{X}}_R = \Big \{({\hat{u}},{\hat{v}}): [0,R] \rightarrow {\mathbb {R}}^2\,\Big | \sqrt{r}{\hat{u}}', \sqrt{r}\,{\hat{v}}', \frac{1}{\sqrt{r}}{\hat{u}}, \sqrt{r}{\hat{v}} \in L^2(0,R), \\&\qquad \qquad \qquad \qquad {\hat{u}}(R) = {\hat{v}}(R) = 0\Big \},\\&\bullet \text { if } R\!=\!\infty , \quad {\hat{X}}_\infty = \left\{ ({\hat{u}},{\hat{v}}): [0,\infty ) \rightarrow {\mathbb {R}}^2\,\Big | \sqrt{r}{\hat{u}}', \sqrt{r}\,{\hat{v}}', \left( \frac{1}{\sqrt{r}} + \sqrt{r}\right) {\hat{u}}, \sqrt{r}{\hat{v}} \in L^2(0,\infty )\right\} . \end{aligned}$$

It is clear that ${\hat{X}}_R$ is a Hilbert space with norm

$$\begin{aligned} \Vert ({\hat{u}},{\hat{v}})\Vert _{R}^2 = \int _0^R \left[ |{\hat{u}}'|^2 + |{\hat{v}}'|^2 + \left( 1 + \frac{1}{r^2}\right) {\hat{u}}^2 +{\hat{v}}^2\right] \,r\,dr. \end{aligned}$$

It is clear that, for $R \in (0,\infty )$,

$$\begin{aligned} {\hat{{\mathscr {E}}}}_R({\hat{u}}, {\hat{v}}) = {\mathscr {E}}_R(u_0 + {\hat{u}}, v_0 + {\hat{v}}) - {\mathscr {E}}_R(u_0, v_0). \end{aligned}$$

We start with some basic remarks on the space ${\hat{X}}_R$ and the functional ${\hat{{\mathscr {E}}}}_R$.

Lemma 3.1

There is some constant $C > 0$ such that for all $({\hat{u}},{\hat{v}}) \in {\hat{X}}_\infty$ we have the following

• (1) Behaviour of ${\hat{u}}$ and ${\hat{v}}$ at $\infty$:

  $$\begin{aligned} |{\hat{u}}(r)|^2\le & {} \frac{C}{r} \int _{\frac{1}{2}}^\infty [|{\hat{u}}'|^2 + |{\hat{u}}|^2]\,s\,ds \text { and }\nonumber \\ |{\hat{v}}(r)|^2\le & {} \frac{C}{r} \int _{\frac{1}{2}}^\infty [|{\hat{v}}'|^2 + |{\hat{v}}|^2]\,s\,ds, \quad \text { for } r \in \left( \frac{1}{2},\infty \right) . \end{aligned}$$

  (3.1)

  In particular, ${\hat{u}}(r), {\hat{v}}(r) \rightarrow 0$ as $r \rightarrow \infty$.

• (2) Behaviour of ${\hat{u}}$ in $(0, \infty )$:

  $$\begin{aligned} |{\hat{u}}(r)|^2 \le C \int _0^r \left[ |{\hat{u}}'|^2 + \frac{1}{s^2} |{\hat{u}}|^2\right] \,s\,ds \quad \text { for } r \in (0,\infty ). \end{aligned}$$

  In particular, ${\hat{u}}(r) \rightarrow 0$ as $r \rightarrow 0$.

• (3) Behaviour of ${\hat{v}}$ at the origin:

  $$\begin{aligned} |{\hat{v}}(r)|^2 \le C\,|\ln r|\,\int _0^\infty [|{\hat{v}}'|^2 + |{\hat{v}}|^2]\,s\,ds \quad \text { for } r \in \left( 0,\frac{1}{2}\right) . \end{aligned}$$

Proof

• (1) See the proof of Strauss’ inequality [37, p. 155].

• (2) For $0< r < r_1$, we estimate

  $$\begin{aligned} |{\hat{u}}^2(r_1)-{\hat{u}}^2(r)| \le 2\int _r^{r_1} |{\hat{u}}(s)||{\hat{u}}'(s)|\,ds\le 2\left( \int _r^{r_1} \frac{{\hat{u}}^2(s)}{s}ds\right) ^{\frac{1}{2}}\left( \int _r^{r_1} |{\hat{u}}'(s)|^2\,s\,ds\right) ^{\frac{1}{2}}. \end{aligned}$$

  This implies that the limit $l:=\lim \nolimits _{r\rightarrow 0} {\hat{u}}^2(r)$ exists. Since $\int \nolimits _0^\infty \frac{{\hat{u}}^2(s)}{s}\,ds<\infty$ we thus have $l=0$, i.e. ${\hat{u}}(r) \rightarrow 0$ as $r \rightarrow 0$. Returning to the above estimate, by the Young inequality we have $2|{\hat{u}}(s)||{\hat{u}}'(s)|\le [|{\hat{u}}'|^2 + \frac{1}{s^2} |{\hat{u}}|^2]\,s$ and we obtain the desired estimate.

• (3) For $R > 0$, consider the minimization problem

  $$\begin{aligned} \alpha (R) = \inf \left\{ \Vert v\Vert _{H^1((R,\infty );r\,dr)}: v \in H^1((R,\infty );r\,dr), v(R) = 1\right\} . \end{aligned}$$

  It is standard that the infimum is achieved and the minimizer $v_*$ is the unique solution of $v_*'' + \frac{1}{r}\,v_*' - v_* = 0$ in (0, R), $v_*(R) = 1$, $v_*(\infty ) = 0$. In terms of special functions, we have $v_* = \frac{K(r)}{K(R)}$, where K is zeroth modified Bessel function of the second kind [1]. It is then readily seen that

  $$\begin{aligned} \alpha (R)^2 = \int _R^\infty [|v_*'|^2 + |v_*|^2]\,r\,dr = r\,v_*'\,v_*\Big |_R^\infty = \frac{R|K'(R)|}{K(R)}. \end{aligned}$$

  As a consequence, we have for all $v\in H^1((0,\infty );r\,dr)$ that

  $$\begin{aligned} \Vert v\Vert _{H^1((0,\infty );r\,dr)} \ge \Vert v\Vert _{H^1((R,\infty );r\,dr)} \ge \alpha (R)\,|v(R)| \text { for all } R \in (0,\infty ). \end{aligned}$$

  Now, since $K(r) = -\ln r + O(1)$ and $K'(r) = -\frac{1}{r} + O(1)$ as $r\rightarrow 0$ [1], we have $\alpha (r) = \frac{1}{\sqrt{|\ln r|}} + O(1)$ as $r \rightarrow 0$, and so, for $r \le \frac{1}{2}$,

  $$\begin{aligned} |v(r)| \le C\,\sqrt{|\ln r|}\,\Vert v\Vert _{H^1((0,\infty );r\,dr)}. \end{aligned}$$

$\square$

Lemma 3.2

Assume that $R\in (0, \infty ]$.

• (1) ${\hat{{\mathscr {E}}}}_R: {\hat{X}}_R \rightarrow {\mathbb {R}}$ is $C^1$ on ${\hat{X}}_R$ with the differential given by

  $$\begin{aligned} D{\hat{{\mathscr {E}}}}_R({\hat{u}}, {\hat{v}})(\xi ,\eta )= & {} \int _0^R \Big \{{\hat{u}}'\,\xi ' + {\hat{v}}'\,\eta ' + \frac{k^2}{r^2}\,{\hat{u}}\,\xi \\&+ \;[Df(u_0 + {\hat{u}}, v_0 + {\hat{v}}) - Df(u_0, v_0)](\xi ,\eta )\Big \}\,r\,dr \end{aligned}$$

  where $({\hat{u}}, {\hat{v}}), (\xi ,\eta ) \in {\hat{X}}_R$. Furthermore $({\hat{u}}, {\hat{v}}) \in {\hat{X}}_R$ is a critical point for ${\hat{{\mathscr {E}}}}_R$ if and only if $(u, v) = (u_0 + {\hat{u}}, v_0 + {\hat{v}})$ is a solution of (1.6) and (1.11).

• (2) ${\hat{{\mathscr {E}}}}_R$ is twice Gâteaux differentiable, meaning here that for every $({\hat{u}}, {\hat{v}}), (\xi ,\eta ) \in {\hat{X}}_R$, the following holds:

  $$\begin{aligned}&D^2{\hat{{\mathscr {E}}}}_R({\hat{u}}, {\hat{v}})(\xi ,\eta )\cdot (\xi ,\eta ):= \frac{d^2}{dt^2}\Big |_{t = 0} {\hat{{\mathscr {E}}}}_R({\hat{u}}+ t\xi , {\hat{v}}+ t\eta )\\&\quad =\int _0^R \Big \{|\xi '|^2 + |\eta '|^2 + \frac{k^2}{r^2}\,\xi ^2 + D^2 f(u_0 + {\hat{u}}, v_0 + {\hat{v}})(\xi ,\eta ) \cdot (\xi ,\eta )\Big \}\,r\,dr. \end{aligned}$$

Proof

The lemma is standard for $R < \infty$. Let us prove it for the more delicate case $R=\infty$.

Step 1: We prove that ${\hat{{\mathscr {E}}}}_\infty ({\hat{u}},{\hat{v}}) < \infty$ for $({\hat{u}}, {\hat{v}}) \in {\hat{X}}_\infty$. To this end, it suffices to prove the following three estimates (for some constant C):

$$\begin{aligned}&\int _0^\infty [|u_0'|^2 + |v_0'|^2]\,r\,dr \le C,\end{aligned}$$

(3.2)

$$\begin{aligned}&\int _0^\infty \frac{u_0\,|{\hat{u}}|}{r}\,dr \le C\,\Big \{\int _0^\infty [|{\hat{u}}'|^2 + |{\hat{u}}|^2]\,r\,dr\Big \}^{1/2} \le C \Vert ({\hat{u}},{\hat{v}})\Vert _{{\hat{X}}_\infty },\end{aligned}$$

(3.3)

$$\begin{aligned}&\int _0^\infty \big |f(u_0 + {\hat{u}},v_0 + {\hat{v}}) - f(u_0,v_0)\big |\,r\,dr \le C\left( 1 + \Vert ({\hat{u}},{\hat{v}})\Vert _{{\hat{X}}_\infty }^4\right) . \end{aligned}$$

(3.4)

${ Proof~of~(3.2)}$: By [26, Proposition 2.3], $u_0, v_0 \in C^2([0,\infty ))$. In addition, by (2.12) and (2.13), $r\,u_0'(r)$ and $r\,v_0'(r)$ have limits as $r \rightarrow \infty$. But as $u_0(r)$ and $v_0(r)$ remain finite as $r \rightarrow \infty$, these limits must be zero, i.e.

$$\begin{aligned} \lim _{r \rightarrow \infty } r\,u_0'(r) = \lim _{r \rightarrow \infty } r\,v_0'(r) = 0. \end{aligned}$$

Multiplying (2.12) and (2.13) by r and integrating on $(r, \infty )$, we obtain

$$\begin{aligned} u_0'(r)&= -2 p_1\,r^{-3} + O(r^{-5}),\end{aligned}$$

(3.5)

$$\begin{aligned} v_0'(r)&= -2q_1\,r^{-3} + O(r^{-5}). \end{aligned}$$

(3.6)

Therefore, $\sqrt{r} u_0'$ and $\sqrt{r}\,v_0'$ belong to $L^2(0,\infty )$ and (3.2) follows.

${ Proof~of~(3.3)}$: By the Sobolev embedding theorem in one dimension, we have that ${\hat{u}}$ and ${\hat{v}}$ are continuous on $(0,\infty )$. Also, by Step 4 in the proof of [26, Proposition 2.3], $\frac{u_0(r)}{r^{|k|}}$ is bounded as $r \rightarrow 0$, and so

$$\begin{aligned} |u_0(r)|\le \frac{Cr^{|k|}}{(1+r)^{|k|}} \quad \text { for all } r \in (0,\infty ). \end{aligned}$$

(3.7)

Estimate (3.3) is readily seen from Lemma 3.1:

$$\begin{aligned} \int _0^\infty \frac{u_0\,|{\hat{u}}|}{r}\,dr\le & {} C\int _0^{\frac{1}{2}}|{\hat{u}}|\, dr+C\int _{\frac{1}{2}}^\infty \frac{|{\hat{u}}|}{r}\,dr\\\le & {} C\,\Big \{\int _0^\infty [|{\hat{u}}'|^2 + |{\hat{u}}|^2]\,r\,dr\Big \}^{1/2}\bigg (\int _0^{\frac{1}{2}} \sqrt{|\ln r|}\, dr+ \int _{\frac{1}{2}}^\infty \frac{1}{r^{3/2}}\,dr\bigg ). \end{aligned}$$

${ Proof~of~(3.4)}$: First, note that $(\frac{s_+}{\sqrt{2}}, -\frac{s_+}{\sqrt{6}})$ is a (global) minimum of f. Thus there is some $\delta > 0$ and $C > 0$ such that for all $|x| \le \delta$ and $|y| \le \delta$, there holds

$$\begin{aligned} 0 \le f\left( \frac{s_+}{\sqrt{2}} + x, -\frac{s_+}{\sqrt{6}} + y\right) - f\left( \frac{s_+}{\sqrt{2}}, -\frac{s_+}{\sqrt{6}}\right) \le C(x^2 + y^2). \end{aligned}$$

(3.8)

Therefore, in view of (3.1), (2.10) and (2.11), there is some sufficiently large $R_1 > 0$ such that

$$\begin{aligned} |f(u_0+{\hat{u}}, v_0+{\hat{v}})-f(u_0,v_0)|\le C\left( \frac{1}{r^4}+{\hat{u}}^2+{\hat{v}}^2\right) \text { in } (R_1, \infty ), \end{aligned}$$

(3.9)

which implies

$$\begin{aligned} \int _{R_1}^\infty |f(u_0+{\hat{u}}, v_0+{\hat{v}})-f(u_0,v_0)|\,r\,dr \le C\left( 1 + \Vert ({\hat{u}},{\hat{v}})\Vert _{{\hat{X}}_\infty }^2\right) . \end{aligned}$$

On the other hand, since f is a quartic polynomial and $u_0$ and $v_0$ are bounded, we have

$$\begin{aligned} |f(u_0 + {\hat{u}}, v_0 + {\hat{v}}) - f(u_0,v_0)| \le C(1 + {\hat{u}}^4 + {\hat{v}}^4) \text { in } (0, \infty ) \end{aligned}$$

for some constant C which depends only on a, b and c. Thus, by the Sobolev embedding theorem (in two dimensions $H^1(B_{R_1})\subset L^4(B_{R_1})$),

$$\begin{aligned} \int _0^{R_1} |f(u_0 + {\hat{u}}, v_0 + {\hat{v}}) - f(u_0,v_0)|\,r\,dr \le C\left( 1 + \Vert ({\hat{u}},{\hat{v}})\Vert _{{\hat{X}}_\infty }^4\right) , \end{aligned}$$

which concludes the proof of (3.4).

Step 2: We prove that ${\hat{{\mathscr {E}}}}_\infty$ is Fréchet differentiable.

Define $A: {\hat{X}}_\infty \rightarrow {\mathbb {R}}$ by

$$\begin{aligned} A(\xi ,\eta )&= \int _0^\infty \Big \{{\hat{u}}'\,\xi ' + {\hat{v}}'\,\eta ' + \frac{k^2}{r^2}\,{\hat{u}}\,\xi + \big [Df(u_0 + {\hat{u}}, v_0 + {\hat{v}}) - Df(u_0, v_0)\big ] (\xi ,\eta )\Big \}\,r\,dr . \end{aligned}$$

Arguing as in the proof of (3.4), we have

$$\begin{aligned} |Df(u_0+{\hat{u}}, v_0+{\hat{v}})-Df(u_0,v_0)|\le C\left( \frac{1}{r^2}+|{\hat{u}}|+|{\hat{v}}|\right) \text { in } (R_1, \infty ), \end{aligned}$$

(3.10)

(for possibly a larger constant $R_1$) and

$$\begin{aligned} |Df(u_0+{\hat{u}}, v_0+{\hat{v}})-Df(u_0,v_0)|\le C(1 + |{\hat{u}}|^3 + |{\hat{v}}|^3) \text { in } (0,\infty ). \end{aligned}$$

(3.11)

As in Step 1, these estimates imply that A is a well-defined continuous linear functional on ${\hat{X}}_\infty$, i.e., $|A(\xi ,\eta )|\le C \Vert (\xi ,\eta )\Vert _{{\hat{X}}_\infty }$.

An easy computation shows that

$$\begin{aligned}&{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}+ \xi , {\hat{v}}+ \eta ) - {\hat{{\mathscr {E}}}}_\infty ({\hat{u}}, {\hat{v}}) - A(\xi ,\eta )\nonumber \\&= \int _0^\infty \left[ u_0'\xi ' + \frac{k^2}{r^2} u_0\xi + h(u_0,v_0)\xi \right] \,r\,dr + \int _0^\infty \big [v_0'\eta ' + g(u_0,v_0)\eta \big ]\,r\,dr \nonumber \\&\qquad + \int _0^\infty P({\hat{u}},{\hat{v}}, \xi ,\eta )\,r\,dr+ O\left( \Vert (\xi ,\eta )\Vert _{{\hat{X}}_\infty }^2\right) , \end{aligned}$$

(3.12)

where

$$\begin{aligned} P({\hat{u}},{\hat{v}}, \xi ,\eta )= & {} f(u_0 + {\hat{u}} + \xi ,v_0 + {\hat{v}} + \eta ) - f(u_0 + {\hat{u}},v_0 + {\hat{v}})\\&- Df(u_0 + {\hat{u}},v_0 + {\hat{v}}) (\xi ,\eta ). \end{aligned}$$

To treat the first two terms on the right hand side of (3.12), recall that $u_0, v_0 \in C^2([0,\infty ))$ [26, Proposition 2.3] and $v_0'(0) = 0$. In particular, $|v_0'(r)| \le Cr$ for some constant C. Thus, using Lebesgue’s dominated convergence theorem, (1.6), (3.5), (3.6), (3.7), (1.11) and Lemma 3.1 (in particular the behavior of $\xi (r)$ and $\eta (r)$ as $r \rightarrow 0$), we compute

$$\begin{aligned} \int _0^\infty \left[ u_0'\xi ' + \frac{k^2}{r^2} u_0\xi + h(u_0,v_0)\xi \right] \,r\,dr \!&=\! \lim _{r \rightarrow \infty } u_0'(r)\,\xi (r) - \lim _{r \rightarrow 0} u_0'(r)\,\xi (r) = 0,\end{aligned}$$

(3.13)

$$\begin{aligned} \int _0^\infty \big [v_0'\eta ' + g(u_0,v_0)\eta \big ]\,r\,dr&= \lim _{r \rightarrow \infty } v_0'(r)\,\eta (r) - \lim _{r \rightarrow 0} v_0'(r)\,\eta (r) = 0. \end{aligned}$$

(3.14)

To treat the remaining integral on the right hand side of (3.12), we note that f is a quartic polynomial, and so

$$\begin{aligned} |P({\hat{u}},{\hat{v}},\xi ,\eta )| \le C (\xi ^2 + \eta ^2)(1 + \xi ^2 + \eta ^2 + {\hat{u}}^2 + {\hat{v}}^2). \end{aligned}$$

Also, in view of Lemma 3.1 and the Sobolev embedding theorem, we have

$$\begin{aligned} \int _0^\infty [|{\hat{u}}|^4 + |{\hat{v}}|^4]\,r\,dr\le & {} C\,\Vert ({\hat{u}},{\hat{v}})\Vert _{{\hat{X}}_\infty }^4,\\ \int _0^\infty [|\xi |^4 + |\eta |^4]\,r\,dr\le & {} C\,\Vert (\xi ,\eta )\Vert _{{\hat{X}}_\infty }^4. \end{aligned}$$

It follows that

$$\begin{aligned} \int _0^\infty |P({\hat{u}},{\hat{v}}, \xi ,\eta )|\,r\,dr = O\left( \Vert (\xi ,\eta )\Vert _{{\hat{X}}_\infty }^2\left( 1 + \Vert (\xi ,\eta )\Vert _{{\hat{X}}_\infty }^2\right) \right) . \end{aligned}$$

(3.15)

Putting together (3.12), (3.13), (3.14) and (3.15), we conclude that ${\hat{{\mathscr {E}}}}_\infty$ is Fréchet differentiable and $D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}, {\hat{v}})= A$. Furthermore, since

$$\begin{aligned} \int _0^R \Big \{u_0'\,\xi ' + v_0'\,\eta ' + \frac{k^2}{r^2}\,u_0\,\xi + Df(u_0, v_0)(\xi ,\eta )\Big \}\,r\,dr=0 \end{aligned}$$

for every $(\xi ,\eta ) \in {\hat{X}}_R$, we deduce that $({\hat{u}}, {\hat{v}}) \in {\hat{X}}_R$ is a critical point for ${\hat{{\mathscr {E}}}}_R$ if and only if $(u, v) = (u_0 + {\hat{u}}, v_0 + {\hat{v}})$ is a solution of (1.6), (1.11).

Step 3: We prove that ${\hat{{\mathscr {E}}}}_\infty$ is twice Gâteaux differentiable.

Define $B: {\hat{X}}_\infty \rightarrow {\mathbb {R}}$ by

$$\begin{aligned} B(\xi ,\eta )&=\int _0^\infty \Big \{|\xi '|^2 + |\eta '|^2 + \frac{k^2}{r^2}\,\xi ^2 + D^2 f(u_0 + {\hat{u}}, v_0 + {\hat{v}})(\xi ,\eta ) \cdot (\xi ,\eta )\Big \}\,r\,dr. \end{aligned}$$

The well-definedness of B can be established similarly as in Step 2 using the estimates

$$\begin{aligned} |D^2 f(u_0 + {\hat{u}}, v_0 + {\hat{v}})|\le & {} C \text { in some interval } (R_1, \infty ),\\ |D^2 f(u_0 + {\hat{u}}, v_0 + {\hat{v}})|\le & {} C(1 + |{\hat{u}}|^2 + |{\hat{v}}|^2) \text { in }(0,\infty ). \end{aligned}$$

Since

$$\begin{aligned} \int _0^{\frac{1}{2}} |{\hat{v}}|^2 |\eta |^2\, rdr\le C \int _0^{\frac{1}{2}} r|\ln r|^2\, dr \int _0^\infty [|\eta '|^2 + |\eta |^2]\,r\,dr , \end{aligned}$$

by Lemma 3.1, we deduce that B satisfies $|B(\xi ,\eta )|\le C \Vert (\xi , \eta )\Vert _{{\hat{X}}_\infty }^2$. The assertion that

$$\begin{aligned} B(\xi ,\eta ) = \frac{d^2}{dt^2}\Big |_{t = 0} {\hat{{\mathscr {E}}}}_\infty ({\hat{u}}+ t\xi , {\hat{v}}+ t\eta )=\frac{d}{dt}\Big |_{t = 0} D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}+ t\xi , {\hat{v}}+ t\eta ) (\xi ,\eta ) \end{aligned}$$

follows from the estimate (for $0< t < 1$)

$$\begin{aligned}&|Df(u_0 + {\hat{u}}+ t\xi , v_0 + {\hat{v}}+ t\eta ) - Df(u_0 + {\hat{u}}, v_0 + {\hat{v}}) - t\,D^2f(u_0 + {\hat{u}}, v_0 + {\hat{v}}) (\xi ,\eta )|\\&\qquad \qquad \qquad \le Ct^2(|\xi |^2 + |\eta |^2)(1 + |{\hat{u}}| + |{\hat{v}}| + |\xi | + |\eta |). \end{aligned}$$

We omit the details.

Step 4: We prove the continuity of the differential $D{\hat{{\mathscr {E}}}}_\infty$ in $({\hat{u}}, {\hat{v}})\in {\hat{X}}_\infty$.

Indeed, since the continuity is a local property, we may assume that $({\hat{u}}, {\hat{v}}), (\tilde{u}, \tilde{v})$ are in a finite ball of radius $\rho$ in ${\hat{X}}_\infty$. Then

$$\begin{aligned}&|Df(u_0 + {\hat{u}}, v_0 + {\hat{v}})- Df(u_0 + \tilde{u}, v_0 + \tilde{v})|\\&\quad \le C(|{\hat{u}}-\tilde{u}| + |{\hat{v}}-\tilde{v}|)(1 + |{\hat{u}}|^2 + |{\hat{v}}|^2 + |\tilde{u}|^2 + |\tilde{v}|^2) \end{aligned}$$

with $C>0$ independent of ${\hat{u}}, {\hat{v}}, \tilde{u}, \tilde{v}$. By Lemma 3.1, we know that $\Vert {\hat{u}}\Vert _\infty , \Vert \tilde{u}\Vert _\infty \le C_\rho$ and

$$\begin{aligned} |{\hat{v}}(r)|, |\tilde{v}(r)|\le C_\rho \bigg (|\ln r|^{1/2} \mathbf{1}_{\left( 0, \frac{1}{2}\right) }(r)+\frac{1}{r} \mathbf{1}_{\left( \frac{1}{2}, \infty \right) }(r)\bigg ), \quad r\in (0, \infty ) \end{aligned}$$

and therefore,

$$\begin{aligned}&\int _0^\infty |Df(u_0 + {\hat{u}}, v_0 + {\hat{v}})- Df(u_0 + \tilde{u}, v_0 + \tilde{v})|^2\, rdr\\&\qquad \quad \le C_\rho \Vert ({\hat{u}}-\tilde{u}, {\hat{v}}-\tilde{v})\Vert _{{\hat{X}}_\infty }^2 \left( 1 +\int _0^{\frac{1}{2}} |\ln r|^4\, rdr+\int _{\frac{1}{2}}^\infty \frac{1}{r^3}\, dr\right) . \end{aligned}$$

We conclude that for every $(\xi , \eta )\in {\hat{X}}_\infty$

$$\begin{aligned} \bigg |[D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}, {\hat{v}})-D{\hat{{\mathscr {E}}}}_\infty (\tilde{u}, \tilde{v})](\xi , \eta )\bigg |\le C_\rho \Vert ({\hat{u}}-\tilde{u}, {\hat{v}}-\tilde{v})\Vert _{{\hat{X}}_\infty } \Vert (\xi , \eta )\Vert _{{\hat{X}}_\infty }, \end{aligned}$$

therefore $D{\hat{{\mathscr {E}}}}_\infty$ is locally Lipschitz in ${\hat{X}}_\infty$. $\square$

Next, we consider coercivity and Palais–Smale properties of ${\hat{{\mathscr {E}}}}_R$.

Lemma 3.3

The following statements hold:

• if $R\in (0, \infty )$, then ${\hat{{\mathscr {E}}}}_R$ is coercive on ${\hat{X}}_R$;

• if $R=\infty$, then ${\hat{{\mathscr {E}}}}_\infty$ is coercive on the closed convex set

  $$\begin{aligned} M_\infty = \{({\hat{u}}, {\hat{v}}) \in {\hat{X}}_\infty : u_0 + {\hat{u}}\ge 0, v_0 + {\hat{v}}\le 0\}, \end{aligned}$$

  (3.16)

  i.e. there exists some $C > 0$ such that

  $$\begin{aligned} {\hat{{\mathscr {E}}}}_\infty ({\hat{u}}, {\hat{v}}) \ge \frac{1}{C}\Vert ({\hat{u}}, {\hat{v}})\Vert _{{\hat{X}}_\infty }^2 - C\quad \text { for all } ({\hat{u}},{\hat{v}}) \in M_\infty . \end{aligned}$$

Proof

Let $({\hat{u}}, {\hat{v}}) \in {\hat{X}}_R$. In the argument below, C denotes various positive constants which are always independent of $({\hat{u}}, {\hat{v}})$.

Let $u_* = \frac{s_+}{\sqrt{2}}$ and $v_* = -\frac{s_+}{\sqrt{6}}$. Since $(u_*,v_*)$ is a minimum of f, we have $f(u_*,v_*) \le f(x,y)$ for all $(x,y) \in {\mathbb {R}}^2$. Also, by (3.8), (2.10) and (2.11) in the case $R=\infty$,

$$\begin{aligned} \int _0^R \big |f(u_0 ,v_0 ) - f(u_*, v_*)\big |\,r\,dr \le C, \end{aligned}$$

(3.17)

(if $R<\infty$ the above inequality is obvious since $u_0$ and $v_0$ are bounded) which implies that

$$\begin{aligned} \int _0^R \big (f(u_0 + {\hat{u}},v_0 + {\hat{v}}) \!-\! f(u_0,v_0)\big )\,r\,dr&\ge \int _0^R \big (f(u_0 + {\hat{u}},v_0 + {\hat{v}}) - f(u_*, v_*)\big )\,r\,dr- C\\&\ge - C. \end{aligned}$$

Case 1: $R<\infty$. From the above estimate, estimates on the finite domain analogous to (3.2), (3.3), and the Poincaré inequality in the disk $B_R$ for ${\hat{u}}, {\hat{v}}\in H^1_0(B_R)$:

$$\begin{aligned} \int _0^R |{\hat{u}}|^2\, rdr\le C \int _0^R |{\hat{u}}'|^2\, rdr, \quad \int _0^R |{\hat{v}}|^2\, rdr\le C \int _0^R |{\hat{v}}'|^2\, rdr, \end{aligned}$$

we deduce that

$$\begin{aligned} {\hat{{\mathscr {E}}}}_R({\hat{u}}, {\hat{v}})\ge C_1 \Vert ({\hat{u}}, {\hat{v}})\Vert _{{\hat{X}}_R}^2 - C_3, \end{aligned}$$

which entails the coercivity of ${\hat{{\mathscr {E}}}}_R$ on ${\hat{X}}_R$.

Case 2: $R=\infty$. Due to the failure of the Poincaré inequality in $H^1({\mathbb {R}}^2)$, the above method does not work for $R=\infty$. We conjecture that ${\hat{{\mathscr {E}}}}_\infty$ is not coercive on ${\hat{X}}_\infty$ and therefore, we prove coercivity only in $M_\infty$. Fix $({\hat{u}}, {\hat{v}}) \in M_\infty$. We would like to improve the estimate on the integral of $|f(u_0 + {\hat{u}},v_0 + {\hat{v}}) - f(u_*, v_*)|$. Let $Q = \{(x,y)\in {\mathbb {R}}^2, x\ge 0,y \le 0\}$. By Lemma 5.1 in the appendix, $f(u_*, v_*) < f(x,y)$ for all $(x,y) \in Q {\setminus } (u_*,v_*)$ and $D^2f(u_*,v_*)$ is positive definite. Also, $\frac{f(x,y)}{x^2 + y^2} \rightarrow \infty$ as $x^2 + y^2 \rightarrow \infty$. Thus, there is some positive constant $\alpha > 0$ such that

$$\begin{aligned} f(x,y) - f(u_*,v_*) \ge \alpha ((x - u_*)^2 + (y - v_*)^2) \quad \text { for all } (x,y) \in Q. \end{aligned}$$

This implies that

$$\begin{aligned} \int _0^\infty \big (f(u_0 + {\hat{u}},v_0 + {\hat{v}}) - f(u_*,v_*)\big )\,r\,dr \ge \alpha \int _0^\infty \big [ |{\hat{u}}+ u_0 - u_*|^2 + |{\hat{v}}+ v_0 - v_*|^2\big ]\,r\,dr.\nonumber \\ \end{aligned}$$

(3.18)

Also, in view of (2.10) and (2.11),

$$\begin{aligned} \int _0^\infty \big [|u_0 - u_*|^2 + |v_0 - v_*|^2\big ]\,r\,dr \le C. \end{aligned}$$

(3.19)

From (3.17), (3.18) and (3.19), we obtain

$$\begin{aligned} \int _0^\infty \big (f(u_0 + {\hat{u}},v_0 + {\hat{v}}) - f(u_0,v_0)\big )\,r\,dr \ge \frac{1}{C} \int _0^\infty [{\hat{u}}^2 + {\hat{v}}^2]\,r\,dr -C. \end{aligned}$$

(3.20)

The desired coercivity of ${\hat{{\mathscr {E}}}}_\infty$ is now readily seen from (3.20), (3.2) and (3.3). $\square$

We recall that a continuously Fréchet differentiable functional I defined on a Banach space X (i.e. $I \in C^1(X, {\mathbb {R}})$) is said to satisfy the Palais–Smale condition if every sequence $\{u_n\}_{n=1}^\infty \subset X$ satisfying $\{I(u_n)\}_{n=1}^\infty$ is bounded and $D\, I (u_n) \rightarrow 0$ in $X'$ (dual of X) is precompact in X, see e.g. [38].

It is not difficult to prove that ${\hat{{\mathscr {E}}}}_R$ satisfies the Palais–Smale condition on ${\hat{X}}_R$ for finite R. It is not clear if this is the case for $R = \infty$; note the restricted coercivity we obtain in Lemma 3.3. We however content ourselves with a milder notion which suffices for our purpose and will be described in the sequel.

Definition 3.4

([38, Section II.12]) Let X be a Banach space, $I \in C^1(X,{\mathbb {R}})$, and M be a closed convex subset of X.

• (i) We say that $x \in M$ is a critical point of I relative to M if

  $$\begin{aligned} \varrho (x) := \sup _{y \in M, \Vert y - x\Vert _X \le 1} D\,I(x) ( x - y ) = 0. \end{aligned}$$

• (ii) We say that I satisfies the Palais–Smale condition on M if every sequence $\{x_n\}_{n=1}^\infty \subset M$ satisfying $\{I(x_n)\}_{n=1}^\infty$ is bounded and $\varrho (x_n) \rightarrow 0$ is precompact in X.

Lemma 3.5

Let $M_\infty$ be as in (3.16) and define

$$\begin{aligned} \varrho ({\hat{u}}, {\hat{v}}) = \sup _{(\xi ,\eta ) \in M_\infty , \Vert (\xi -{\hat{u}}, \eta -{\hat{v}}) \Vert _{{\hat{X}}_\infty } \le 1 } D\,{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}, {\hat{v}})({\hat{u}}- \xi , {\hat{v}}-\eta ). \end{aligned}$$

(3.21)

If $({\hat{u}}_m, {\hat{v}}_m) \in M_\infty$ converges weakly in ${\hat{X}}_\infty$ to $({\hat{u}},{\hat{v}})$ and if $\varrho ({\hat{u}}_m, {\hat{v}}_m) \rightarrow 0$ as $m \rightarrow \infty$, then $(u_0 + {\hat{u}}, v_0 + {\hat{v}})$ satisfies (1.6). In particular, $({\hat{u}},{\hat{v}}) \in M_\infty$ is a critical point of ${\hat{{\mathscr {E}}}}_\infty$ relative to $M_\infty$ if and only if $(u_0 + {\hat{u}}, v_0 + {\hat{v}})$ satisfies (1.6) and (1.11).

Proof

It is enough to show the first assertion. In view of the Sobolev embedding theorem in one dimension, we can assume without loss of generality that $({\hat{u}}_m, {\hat{v}}_m)$ converges uniformly on compact subsets of $(0,\infty )$ to $({\hat{u}}, {\hat{v}})$.

Let $\varrho _m = \varrho ({\hat{u}}_m,{\hat{v}}_m)$. By definition, we have

$$\begin{aligned} D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}_m,{\hat{v}}_m)( {\hat{u}}_m - {\tilde{u}}, {\hat{v}}_m - {\tilde{v}}) \le \varrho _m \text { for all } ({\tilde{u}},{\tilde{v}}) \in M_\infty , \Vert ({\tilde{u}} - {\hat{u}}_m, {\tilde{v}} - {\hat{v}}_m)\Vert _{{\hat{X}}_\infty } \le 1.\nonumber \\ \end{aligned}$$

(3.22)

In particular, we have

$$\begin{aligned} D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}_m,{\hat{v}}_m)( \xi , \eta ) \le \varrho _m\Vert (\xi ,\eta )\Vert _{{\hat{X}}_\infty } \quad \text { for all } (\xi ,\eta ) \in {\hat{X}}_\infty : \xi \le 0, \eta \ge 0\quad \text { in } (0,\infty ). \end{aligned}$$

Since $({\hat{u}}_m, {\hat{v}}_m)$ converges weakly to $({\hat{u}}, {\hat{v}})$ and $\varrho _m \rightarrow 0$, we deduce that

$$\begin{aligned} D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}},{\hat{v}})( \xi , \eta ) \le 0\quad \text { for all } (\xi ,\eta ) \in {\hat{X}}_\infty : \xi \le 0, \eta \ge 0\quad \text { in } (0,\infty ). \end{aligned}$$

This implies that $u := u_0 + {\hat{u}}$ and $v := v_0 + {\hat{v}}$ satisfy in the weak sense the differential inequalities

$$\begin{aligned} u'' + \frac{1}{r}u' - \frac{k^2}{r^2}u&\le h(u,v),\end{aligned}$$

(3.23)

$$\begin{aligned} v'' + \frac{1}{r} v'&\ge g(u,v) \text { in } (0,\infty ). \end{aligned}$$

(3.24)

We claim that if $u > 0$ in any interval $(r_1, r_2) \subset (0,\infty )$ then the first equation of (1.6) [i.e. equality in (3.23)] holds in $(r_1,r_2)$. Indeed, if $\xi \in C_c^\infty (r_1,r_2)$, then in view of the local uniform convergence of ${\hat{u}}_m$ to ${\hat{u}}$, there is some $\epsilon _0 > 0$ such that $({\hat{u}}_m + t\,\xi , {\hat{v}}_m)$ belongs to $M_\infty$ for all $|t| < \epsilon _0$ and for all sufficiently large m. It thus follows from (3.22) that, there is some $t \in (0,\epsilon _0)$ such that

$$\begin{aligned} D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}_m,{\hat{v}}_m)(\pm t\xi , 0) \le \varrho _m\Vert (t\xi ,0)\Vert _{{\hat{X}}_\infty }. \end{aligned}$$

As above, this implies that

$$\begin{aligned} D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}},{\hat{v}})(\pm t\xi , 0) \le 0, \end{aligned}$$

which implies that $D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}},{\hat{v}})( \xi ,0) = 0$. Since $\xi$ is arbitrary, the claim follows.

Similarly, if $v < 0$ in any interval $(r_1, r_2) \subset (0,\infty )$, then the second equation of (1.6) holds in that interval.

Since u is continuous, we can write $\{r: u(r) > 0\} = \cup _{j \in \Lambda } (\alpha _j, \beta _j)$ of at most countably many mutually disjoint open intervals. As argued above, the first equation of (1.6) holds on each interval $(\alpha _j, \beta _j)$. (Initially, it holds in the weak sense, but since u and v are Hölder continuous (in view of the Sobolev embedding theorem in one dimension), it holds in the classical sense.) Furthermore, $u(\alpha _j) = 0$ if $\alpha _j > 0$, and $u(\beta _j) = 0$ if $\beta _j < \infty$. Since $h(u,v) = u\,c_1$ for some continuous function $c_1$, the Hopf lemma implies that

$$\begin{aligned} u'(\alpha _j)> 0\text { if } \alpha _j > 0 \quad \text { and }\quad u'(\beta _j)< 0 \quad \text { if }\beta _j < \infty . \end{aligned}$$

(3.25)

Recall that $u' = 0$ a.e. in the set $\{u = 0\}$. Now, for any $\xi \in C_c^\infty (0,\infty )$ and $\xi \ge 0$, we have in view of (3.23) that

$$\begin{aligned} 0\le & {} \int _0^\infty \left[ u'\,\xi ' + \frac{1}{r^2} u \xi + h(u,v) \xi \right] \,rdr\\= & {} \sum _{j \in \Lambda } \int _{\alpha _j}^{\beta _j} \left[ u'\,\xi ' + \frac{1}{r^2} u \xi + h(u,v) \xi \right] \,rdr\\= & {} \sum _{j \in \Lambda } \big [\beta _j\,u'(\beta _j)\,\xi (\beta _j) - \alpha _j\,u'(\alpha _j)\,\xi (\alpha _j)\big ], \end{aligned}$$

where in the first equality, we have used $h(u,v) = 0$ wherever $u = 0$. By (3.25), if there is some j such that $\alpha _j$ or $\beta _j$ is non-zero and finite, the last sum is negative if $\xi$ is chosen to be positive thereof. We thus conclude that the $\alpha _j$ and $\beta _j$’s are either zero or infinite, i.e. $u > 0$ in $(0,\infty )$. We hence deduce that the first equation of (1.6) holds in $(0,\infty )$.

The negativity of v and the validity of the second equation of (1.6) can be demonstrated similarly, keeping in mind that

$$\begin{aligned} g(u,v) \ge v\left( - a^2 - \frac{1}{\sqrt{6}}b^2\,v + c^2(u^2 + v^2)\right) =: v\,c_2, \end{aligned}$$

and in particular, $g(u,v) \ge 0$ wherever $v = 0$. We omit the details. $\square$

In the following lemma we prove that ${\hat{{\mathscr {E}}}}_R$ satisfies the Palais–Smale condition.

Lemma 3.6

For $R \in (0,\infty )$, ${\hat{{\mathscr {E}}}}_R$ satisfies the Palais–Smale condition on ${\hat{X}}_R$. For $R = \infty$, ${\hat{{\mathscr {E}}}}_\infty$ satisfies the Palais–Smale condition on the closed convex set $M_\infty$ defined in (3.16).

Proof

The result is standard for $R < \infty$. Consider the case $R = \infty$. Let $({\hat{u}}_m, {\hat{v}}_m) \in M_\infty$ be a Palais–Smale sequence for ${\hat{{\mathscr {E}}}}_\infty$, i.e. ${\hat{{\mathscr {E}}}}_\infty ({\hat{u}}_m, {\hat{v}}_m)$ is bounded and $\varrho ({\hat{u}}_m, {\hat{v}}_m) \rightarrow 0$, where $\varrho$ is defined in (3.21). We need to show that $({\hat{u}}_m, {\hat{v}}_m)$ has a convergent subsequence in ${\hat{X}}_\infty$.

By Lemma 3.3, the sequence $({\hat{u}}_m, {\hat{v}}_m)$ is bounded in ${\hat{X}}_\infty$ and so we can assume without loss of generality that $({\hat{u}}_m, {\hat{v}}_m)$ converges weakly in ${\hat{X}}_\infty$ to some $({\hat{u}}, {\hat{v}})$. By the Sobolev embedding theorem (in one and two dimensions), we can also assume that $({\hat{u}}_m, {\hat{v}}_m)$ converges to $({\hat{u}}, {\hat{v}})$, uniformly on compact subsets of $(0,\infty )$ and strongly in $L^p((0,R);r\,dr)$ for any $R < \infty$ and $1 \le p < \infty$.

By Lemma 3.5, $u_1 : = u_0 + {\hat{u}}$ and $v_1 := v_0 + {\hat{v}}$ is a solution to (1.6), (1.11). By working with $(u_1, v_1)$ instead of $(u_0, v_0)$ and with the sequence $({\hat{u}}_m - {\hat{u}}, {\hat{v}}_m - {\hat{v}})$ instead of $({\hat{u}}_m, {\hat{v}}_m)$, we can assume for simplicity that ${\hat{u}}= {\hat{v}}= 0$.

Let

$$\begin{aligned} V_m := Df(u_0 + {\hat{u}}_m, v_0 + {\hat{v}}_m) - Df(u_0, v_0). \end{aligned}$$

As in the proof of Lemma 3.3, let $u_* = \frac{s_+}{\sqrt{2}}$ and $v_* = -\frac{s_+}{\sqrt{6}}$ and note that $D^2f(u_*,v_*)$ is (strictly) positive definite, which implies that there are $\alpha , \delta > 0$ such that

$$\begin{aligned}&[Df(x,y) - Df(x',y')] (x - x',y - y')\\&\quad \ge \alpha [(x - x')^2 + (y - y')^2] \quad \forall ~(x,y), (x',y') \in B_\delta (u_*,v_*). \end{aligned}$$

Thus in view of (2.10), (2.11) and Strauss’ inequality (3.1) applied to the bounded sequence $({\hat{u}}_m, {\hat{v}}_m)$ in ${\hat{X}}_\infty$, there is some large $R_2 > 0$ (independent of m) such that

$$\begin{aligned} V_m ({\hat{u}}_m, {\hat{v}}_m) \ge \alpha [|{\hat{u}}_m|^2 + |{\hat{v}}_m|^2] \quad \text { in } (R_2,\infty ). \end{aligned}$$

(3.26)

On the other hand, note that $(t{\hat{u}}_m,t{\hat{v}}_m) \in M_\infty$ for all $t \in [0,1]$. As $({\hat{u}}_m,{\hat{v}}_m)$ is bounded in ${\hat{X}}_\infty$, we can select some $t_0 \in (0,1)$ independent of m such that

$$\begin{aligned} \Vert ({\hat{u}}_m - t_0{\hat{u}}_m, {\hat{v}}_m - t_0{\hat{v}}_m)\Vert _{{\hat{X}}_\infty } \le 1. \end{aligned}$$

Then it follows from (3.21) that

$$\begin{aligned} \varrho ({\hat{u}}_m,{\hat{v}}_m) \ge D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}_m, {\hat{v}}_m) ({\hat{u}}_m - t_0{\hat{u}}_m, {\hat{v}}_m - t_0{\hat{v}}_m), \end{aligned}$$

which together with (3.26) implies that

$$\begin{aligned} \frac{1}{1 - t_0}\varrho ({\hat{u}}_m,{\hat{v}}_m)\ge & {} D{\hat{{\mathscr {E}}}}_\infty ({\hat{u}}_m,{\hat{v}}_m)({\hat{u}}_m,{\hat{v}}_m) \\= & {} \int _0^\infty \Big \{|{\hat{u}}_m'|^2 + |{\hat{v}}_m'|^2 + \frac{k^2}{r^2}\,|{\hat{u}}_m|^2 + V_m ({\hat{u}}_m, {\hat{v}}_m)\Big \}\,r\,dr\\\ge & {} \int _0^\infty \Big \{|{\hat{u}}_m'|^2 + |{\hat{v}}_m'|^2 + \frac{k^2}{r^2}\,|{\hat{u}}_m|^2 + \alpha |{\hat{u}}_m|^2 + \alpha |{\hat{v}}_m|^2\Big \}\,r\,dr\\&+ \int _0^{R_2} \Big \{-\alpha |{\hat{u}}_m|^2 - \alpha |{\hat{v}}_m|^2 + V_m ({\hat{u}}_m, {\hat{v}}_m)\Big \}\,r\,dr. \end{aligned}$$

On the other hand, by the strong convergence of $({\hat{u}}_m, {\hat{v}}_m)$ to (0, 0) in $L^p((0,R_2);r\,dr)$ and the estimate (3.11), we see that

$$\begin{aligned} \lim _{m \rightarrow \infty }\int _0^{R_2} \Big \{-\alpha |{\hat{u}}_m|^2 - \alpha |{\hat{v}}_m|^2 + V_m ({\hat{u}}_m, {\hat{v}}_m)\Big \}\,r\,dr = 0. \end{aligned}$$

Recalling that $\varrho _m \rightarrow 0$, we obtain that $({\hat{u}}_m, {\hat{v}}_m)$ converges in ${\hat{X}}_\infty$ to $0 = ({\hat{u}}, {\hat{v}})$. $\square$

The following result is a consequence of the above lemmas and a variant of the mountain pass theorem [38, Theorem II.12.8].

Lemma 3.7

• (a) Let $R\in (0, \infty )$. If all critical points in ${\hat{X}}_R$ of ${\hat{{\mathscr {E}}}}_R$ are strictly stable, then ${\hat{{\mathscr {E}}}}_R$ has a unique critical point.

• (b) Let $R\in (0, \infty ]$. If all critical points $({\hat{u}}, {\hat{v}})$ of ${\hat{{\mathscr {E}}}}_R$ satisfying $u_0 + {\hat{u}} \ge 0$ and $v_0 + {\hat{v}} \le 0$ are strictly stable, then ${\hat{{\mathscr {E}}}}_R$ has a unique critical point satisfying $u_0 + {\hat{u}} \ge 0$ and $v_0 + {\hat{v}} \le 0$.

In view of the above result, to prove uniqueness in Theorem 1.7, it suffices to establish (strict) stability at relevant critical points (u, v). It is readily seen that, for $R \in (0,\infty ]$,

$$\begin{aligned} D^2{\hat{{\mathscr {E}}}}_R(u - u_0, v - v_0) (\xi , \eta )\cdot (\xi , \eta ) = B(\xi , \eta ) , \end{aligned}$$

where B is given by (1.18).

Proposition 3.8

Let $a^2, b^2, c^2>0$ be such that $b^4\le 3a^2c^2$. Assume that $R \in (0,\infty ]$. Let $k\in {\mathbb {Z}}{\setminus }\{0\}$ and (u, v) be a solution of (1.6) and (1.11) with $u>0$ and $v<0$. Then (u, v) is strictly stable for ${\hat{{\mathscr {E}}}}_R$ in the sense that $B(\xi ,\eta )>0$ for every nonzero $(\xi ,\eta ) \in {\hat{X}}_R$.

Proof

We will only prove the case $R = \infty$. (The case $R<\infty$ is simpler since the asymptotical behavior at infinity can be dropped.)

Recall from [25, Proposition 2.2], (2.10), (2.11), (3.5), (3.6) and (3.7) that

$$\begin{aligned}&u(r) = O(r^{|k|}), u'(r) = O(r^{|k|-1}), v'(r) = O(r)\quad \text { as } r \rightarrow 0,\end{aligned}$$

(3.27)

$$\begin{aligned}&u'(r) = O(r^{-3}), v'(r) = O(r^{-3}) \quad \text { as } r \rightarrow \infty . \end{aligned}$$

(3.28)

Recalling $h = \frac{\partial f}{\partial u}$ and $g = \frac{\partial f}{\partial v}$, we obtain the estimate

$$\begin{aligned}&h(u,v) = O(r^{|k|}), g(u,v) = O(1) \quad \text { as } r \rightarrow 0,\end{aligned}$$

(3.29)

$$\begin{aligned}&h(u,v) = O(r^{-2}), g(u,v) = O(r^{-2}) \quad \text { as } r \rightarrow \infty . \end{aligned}$$

(3.30)

Fix $(\xi ,\eta ) \in {\hat{X}}_\infty$. Since $u > 0$ and $v < 0$, we can write $\xi =u {\tilde{\xi }}$ and $\eta =v {\tilde{\eta }}$ where ${\tilde{\xi }}, {\tilde{\eta }} \in H^1_{loc}(0,\infty )$. By Lemma 3.1 and (3.27), we have

$$\begin{aligned}&{\tilde{\xi }}(r) = o(r^{-|k|}), {\tilde{\eta }}(r) = O(|\ln r|^{1/2})\quad \text { as } r \rightarrow 0,\end{aligned}$$

(3.31)

$$\begin{aligned}&{\tilde{\xi }}(r) = O(r^{-1/2}), {\tilde{\eta }}(r) = O(r^{-1/2})\quad \text { as } r\rightarrow \infty . \end{aligned}$$

(3.32)

We compute, using (1.6),

$$\begin{aligned} \int _{1/m}^m \left[ |\xi '|^2 + \frac{k^2}{r^2}\,\xi ^2\right] \,r\,dr= & {} \int _{1/m}^m \left[ u^2\,|{\tilde{\xi }}'|^2 + u'\,(u\,{\tilde{\xi }}^2)' + \frac{k^2}{r^2} \,u^2\,{\tilde{\xi }}^2\right] \,r\,dr\\= & {} r\,u'\,u\,{\tilde{\xi }}^2\Big |_{1/m}^m + \int _{1/m}^m \Big [ u^2\,|{\tilde{\xi }}'|^2 - h(u,v)\,u\,{\tilde{\xi }}^2\Big ]\,r\,dr\\= & {} o(1)+ \int _{1/m}^m \Big [ u^2\,|{\tilde{\xi }}'|^2 - h(u,v)\,u\,{\tilde{\xi }}^2\Big ]\,r\,dr \quad \text {as} \quad m\rightarrow \infty , \end{aligned}$$

where we have used (3.27), (3.28), (3.31) and (3.32) in the last identity. Therefore, by monotone and dominated convergence theorems, and (3.27), (3.28), (3.29) and (3.30), since $(\xi ,\eta ) \in {\hat{X}}_\infty$,

$$\begin{aligned} \int _0^\infty \left[ |\xi '|^2 + \frac{k^2}{r^2}\,\xi ^2\right] \,r\,dr&= \lim _{m\rightarrow \infty }\int _{1/m}^m \left[ |\xi '|^2 + \frac{k^2}{r^2}\,\xi ^2\right] \,r\,dr\\&=\int _{0}^\infty \left[ u^2\,|{\tilde{\xi }}'|^2 - h(u,v)\,u\,{\tilde{\xi }}^2\right] \,r\,dr. \end{aligned}$$

Likewise,

$$\begin{aligned} \int _0^\infty |\eta '|^2\,r\,dr&= \int _{0}^\infty \Big [ v^2\,|{\tilde{\eta }}'|^2 - g(u,v)\,v\,{\tilde{\eta }}^2\Big ]\,r\,dr. \end{aligned}$$

We hence obtain

$$\begin{aligned} B(\xi , \eta )= & {} \int _{0}^\infty \bigg \{ u^2| {\tilde{\xi }}'|^2 + v^2|{\tilde{\eta }}'|^2 \\&+ \left( - \frac{b^2}{v \sqrt{6}} (v^2 +u^2) + 2c^2 v^2 \right) \eta ^2 + 2c^2 u^2 \xi ^2+ 4 u \xi \eta \left( \frac{b^2}{\sqrt{6}} + c^2 v \right) \bigg \}\, rdr. \end{aligned}$$

Note that $B(\xi , \eta ) > 0$ for $(\xi ,\eta ) \not \equiv 0$, provided that

$$\begin{aligned} \frac{2\left( \frac{b^2}{\sqrt{6}} + c^2 v \right) ^2}{c^2}\le & {} \left( - \frac{b^2}{v \sqrt{6}} (v^2 +u^2) + 2c^2 v^2 \right) \\\Longleftrightarrow & {} 2b^2\bigg (\frac{b^2}{\sqrt{6}c^2} +v\bigg )+2b^2v \le - \frac{b^2}{v} (v^2 +u^2) \end{aligned}$$

which holds true in $(0, \infty )$ because the above LHS is negative while the RHS is positive due to the inequalities $v<0$ and $b^2+\sqrt{6}c^2v\le 0$ for $b^4\le 3a^2c^2$ (see (2.1)). $\square$

We conclude the section with the proof of Theorem 1.7.

Proof of Theorem 1.7

The result is a consequence of Lemma 3.7(b) and Proposition 3.8.

4 Stability for $k=\pm 1$

In this section we provide the proof of Theorem 1.4 regarding the sign of the second variation ${\mathscr {L}}[Q] (P)$ at k-radially symmetric solutions Q in direction $P \in H^1_0(B_R, {\mathscr {S}}_0)$. Note that for $R = \infty$, $H^1_0({\mathbb {R}}^2, {\mathscr {S}}_0) \equiv H^1({\mathbb {R}}^2, {\mathscr {S}}_0)$. Recall from (1.17) that, for $P \in H_0^1(B_R, {\mathscr {S}}_0)$,

$$\begin{aligned} {{\mathscr {L}}}[Q](P) =\int _{B_R}\Big \{|\nabla P|^2-{a^2}|P|^2-2b^2\mathrm{tr}(P^2 Q)+{c^2}\left( |Q|^2|P|^2+2|\mathrm{tr}(QP)|^2\right) \Big \}\,dx. \end{aligned}$$

4.1 Basis decomposition

In order to prove Theorem 1.4 we use, as in [26], the following basis decomposition. We define $\{ e_i\}_{i=1}^3$ to be the standard basis in ${\mathbb {R}}^3$ and denote, for $\varphi \in [0,2\pi )$ and $k \ne 0$,

$$\begin{aligned} n=n(\varphi )=\left( \cos \left( {\textstyle \frac{k}{2}} \varphi \right) , \sin \left( {\textstyle \frac{k}{2}} \varphi \right) , 0\right) , \, m=m(\varphi )=\left( -\sin \left( {\textstyle \frac{k}{2}} \varphi \right) ,\cos \left( {\textstyle \frac{k}{2}} \varphi \right) ,0\right) . \end{aligned}$$

We endow the space ${\mathscr {S}}_0$ of Q-tensors with the Frobenius scalar product

$$\begin{aligned}Q\cdot {\tilde{Q}}=\mathrm{tr}(Q{\tilde{Q}})\end{aligned}$$

and for any $\varphi \in [0, 2\pi )$, we define the following orthonormal basis in ${\mathscr {S}}_0$:

$$\begin{aligned} E_0= & {} \sqrt{\frac{3}{2}}\left( e_3\otimes e_3-\frac{1}{3}I_3\right) ,\\ E_1= & {} E_1(\varphi )=\sqrt{2}\left( n\otimes n-\frac{1}{2}I_2\right) ,\,E_2=E_2(\varphi )=\frac{1}{\sqrt{2}}\left( n\otimes m+m\otimes n\right) , \\ E_3= & {} \frac{1}{\sqrt{2}}(e_1 \otimes e_3+e_3\otimes e_1),\, E_4=\frac{1}{\sqrt{2}}\left( e_2\otimes e_3+e_3\otimes e_2\right) . \end{aligned}$$

Obviously, only $E_1$ and $E_2$ depend on $\varphi$ and we have

$$\begin{aligned} \frac{\partial E_1}{\partial \varphi }=kE_2 \quad \text {and} \quad \frac{\partial E_2}{\partial \varphi }=-kE_1. \end{aligned}$$

(4.1)

The above basis $\{E_0, \ldots , E_4\}$ is constructed so that at a point $Q_*=s_+\left( n\otimes n-\frac{1}{3}I_3\right)$ with $n\in \mathbb {S}^1\times \{0\}$, tensor $E_2$ is along the direction of the tangent line to ${\mathscr {S}}^{lim}_*$ (see (1.14)), while tensors $E_0$, $E_1$, $E_3$ and $E_4$ are along the normal directions to the tangent line.

It is clear that any $P \in H_{loc}^1({\mathbb {R}}^2, {\mathscr {S}}_0)$ can be represented as

$$\begin{aligned} P(x)= \sum _{i=0}^4 w_i(x) E_i, \quad x = r(\cos \varphi , \sin \varphi ) \in {\mathbb {R}}^2, \end{aligned}$$

with $w_i=P\cdot E_i$ for $i=0,\ldots , 4$. We note although n and m may not be smooth as a function of x, $E_i$ are smooth away from the origin. Then the second variation becomes

$$\begin{aligned} {\mathscr {L}}[Q](P)= & {} \int _{B_R} \bigg \{\sum _{i=0}^4 |\nabla w_i|^2+\frac{k^2}{r^2}(w_1^2+w_2^2)+\frac{2k}{r^2}\left( w_1 \frac{\partial w_2}{\partial \varphi }-w_2 \frac{\partial w_1}{\partial \varphi }\right) \nonumber \\&+ \left( -{a^2} + c^2 (u^2+v^2) \right) \sum _{i=0}^4 |w_i |^2+2 {c^2}\left( v w_0 + u w_1 \right) ^2 \nonumber \\&-\frac{2b^2}{\sqrt{6}} \left( v ( w_0^2 - w_1^2 - w_2^2) - 2 u w_0 w_1\right) \nonumber \\&-\frac{2b^2}{\sqrt{6}}\bigg (\frac{\sqrt{3}}{2}u(w_3^2-w_4^2)\cos (k\varphi )\!+\!\sqrt{3}uw_3w_4\sin (k\varphi )\!+\!\frac{1}{2}v(w_3^2+w_4^2) \bigg )\bigg \}\, dx.\nonumber \\ \end{aligned}$$

(4.2)

We note that components $\{w_0, w_1, w_2\}$ and $\{ w_3, w_4\}$ in (4.2) are not mixed and therefore we can separately study the sign of ${\mathscr {L}}[Q](P)$ in the spaces

$$\begin{aligned}&V_1 = \{ P \in H_0^1(B_R, {\mathscr {S}}_0) \, : \, P \cdot E_3 = P \cdot E_4 =0 \},\\&V_2 = \{ P \in H_0^1(B_R, {\mathscr {S}}_0) \, : \, P = w_3 (x) E_3 + w_4(x) E_4; \, w_3, w_4 \in H_0^1(B_R) \}. \end{aligned}$$

It is clear that $H_0^1(B_R, {\mathscr {S}}_0) =V_1 \oplus V_2$. Furthermore, if P belongs to $H^1_0(B_R, {\mathscr {S}}_0)$, then so do its (direct sum) projections onto $V_1$ and $V_2$.

4.2 Stability in the space $V_1$

We start with the result about stability of ${\mathscr {L}}[Q]$ in $V_1$.

Proposition 4.1

Let $a^2 \ge 0$, $b^2, c^2 >0$ be fixed constants, $R \in (0, \infty ]$, and $k =\pm 1$. Let (u, v) be a solution of (1.6) on (0, R) under the boundary condition (1.11) such that $u > 0$ and $v < 0$ and assume that (u, v) is stable with respect to ${\mathscr {E}}_R$ (i.e. (1.19) holds). Let Q be k-radially symmetric solution Q of (1.2) (on $B_R)$ and (1.9) given by (1.4). Then ${\mathscr {L}}[Q](P) \ge 0$ for any $P \in V_1$.

We will use the following lemma whose simple proof we omit.

Lemma 4.2

Let $|k| = 1$ and $P = w_0\,E_0 + w_1\,E_1 + w_2\,E_2 \in V_1$. If we write

$$\begin{aligned} w_l (r,\varphi ) = \sum _{m=-\infty }^\infty (w_{l,m}(r) + i\,{\hat{w}}_{l,m}(r)) e^{i m \varphi } \quad \text { with }\, \, l=0,1,2, \end{aligned}$$

(4.3)

Then

$$\begin{aligned}&\sqrt{r} (|w_{l,m}'| + |w_{l,m}|), \sqrt{r} (|{\hat{w}}_{l,m}'| + |{\hat{w}}_{l,m}|)\in L^2(0,R) \text { for all } (l,m),\\&\frac{1}{\sqrt{r}} |w_{l,m}|, \frac{1}{\sqrt{r}} |{\hat{w}}_{l,m}| \in L^2(0,R) \text { for all } (l,m) \notin \{(0,0), (1,\pm 1), (2,\pm 1)\},\\&\frac{1}{\sqrt{r}} |kw_{1,m} - m {\hat{w}}_{2,m}|, \frac{1}{\sqrt{r}} |k w_{2,m} + m {\hat{w}}_{1,m}| \in L^2(0,R) \text { for } |m| = 1. \end{aligned}$$

Furthermore, for each $m \in {\mathbb {Z}}$,

$$\begin{aligned} \sum _{l = 0}^2 (w_{l,m}(r) + i\,{\hat{w}}_{l,m}(r)) e^{i m \varphi }\,E_l \in V_1. \end{aligned}$$

Proof of Proposition 4.1

Let us first show that ${\mathscr {L}}[Q](P) \ge 0$ for all $P \in V_1$. By a standard density argument, we can assume without loss of generality that $P \in V_1 \cap C_c^\infty (B_R {\setminus } \{0\})$. (Here we have used the fact that a point has zero Newtonian capacity in two dimensions.)

We write $x=re^{i\varphi }=(r\cos \varphi , r\sin \varphi )$ and $P = w_0 E_0 + w_1 E_1 + w_2 E_2$ as in Sect. 4.1. By (4.2),

$$\begin{aligned} {\mathscr {L}}[Q](P)= & {} \int _0^R \int _0^{2\pi } \bigg \{\sum _{l=0}^2 |\partial _r w_l |^2 +\frac{1}{r^2} \left( |\partial _\varphi w_0|^2+ |\partial _\varphi w_1 - k w_2|^2 + |\partial _\varphi w_2 + k w_1|^2 \right) \\&+ \left( -{a^2} + c^2 (u^2+v^2) \right) \sum _{l=0}^2 |w_l |^2+2 {c^2}\left( v w_0 + u w_1 \right) ^2 \\&-\frac{2b^2}{\sqrt{6}} \left( v ( w_0^2 - w_1^2 - w_2^2) - 2 u w_0 w_1 \right) \bigg \}\, rdr\,d\varphi . \end{aligned}$$

Now, we Fourier decompose $w_l$’s as in (4.3). By Lemma 4.2,

$$\begin{aligned} P_m := \sum _{l = 0}^2 (w_{l,m}(r) + i\,{\hat{w}}_{l,m}(r)) e^{i m \varphi }\,E_l \in V_1. \end{aligned}$$

(4.4)

Furthermore, a direct computation shows that

$$\begin{aligned} {\mathscr {L}}[Q](P)= \sum _{m=-\infty }^\infty {\mathscr {L}}[Q](P_m), \end{aligned}$$

(4.5)

and

$$\begin{aligned} {\mathscr {L}}[Q](P_m)= & {} 2\pi \int _0^R \Biggl \{ \sum _{l=0}^2 \big [| w_{l,m}'|^2+|{\hat{w}}_{l,m}'|^2\big ] + \frac{m^2}{r^2}\big [|w_{0,m}|^2 +|{\hat{w}}_{0,m}|^2\big ] \\&+\, \frac{1}{r^2} \Big [| m w_{1,m} - k {\hat{w}}_{2,m}|^2 + | k w_{1,m} - m {\hat{w}}_{2,m}|^2\\&+\,| m {\hat{w}}_{1,m} + k w_{2,m}|^2 + | k {\hat{w}}_{1,m} + m w_{2,m}|^2 \Big ] \\&+\, \left( -a^2 -\frac{2}{\sqrt{6}}b^2 v+c^2\left( u^2+ 3 v^2\right) \right) (|w_{0,m}|^2 +|{\hat{w}}_{0,m}|^2)\\&+\, \left( -a^2 +\frac{2}{\sqrt{6}}b^2 v+c^2\left( 3 u^2+ v^2\right) \right) (|w_{1,m}|^2 +|{\hat{w}}_{1,m}|^2)\\&+\, 4 u (w_{0,m} w_{1,m} + {\hat{w}}_{0,m} {\hat{w}}_{1,m}) \left[ \frac{b^2}{\sqrt{6}} + c^2 v \right] \\&+\, \left( -a^2 + \frac{2}{\sqrt{6}}b^2 v+c^2\left( u^2+ v^2\right) \right) (| w_{2,m} |^2 + | {\hat{w}}_{2,m} |^2) \Biggr \} r\, dr. \end{aligned}$$

Now, observe that if we define

$$\begin{aligned} \mathcal{P}_m (w_0, w_1, w_2)= & {} \int _0^R \Biggl \{ | w_0'|^2 + | w_1'|^2 + | w'_2 |^2 + \frac{m^2 }{r^2}|w_0|^2\\&+\, \frac{1}{r^2} \left( \big | |m| w_1 - |k| w_2\big |^2 + \big | |k| w_1 - |m| w_2\big |^2 \right) \\&+\, \left( -a^2 -\frac{2}{\sqrt{6}}b^2 v+c^2\left( u^2+ 3 v^2\right) \right) |w_0|^2\\&+ \,\left( -a^2 +\frac{2}{\sqrt{6}}b^2 v+c^2\left( 3u^2+ v^2\right) \right) |w_1|^2 + 4 u w_0 w_1 \left[ \frac{b^2}{\sqrt{6}} + c^2 v \right] \\&+ \left( -a^2 + \frac{2}{\sqrt{6}}b^2 v+c^2\left( u^2+ v^2\right) \right) | w_2 |^2 \Biggr \} r\, dr, \end{aligned}$$

then

$$\begin{aligned} \frac{1}{2\pi }{\mathscr {L}}[Q](P_m)= & {} \mathcal{P}_m(\mathrm{sign}(m)w_{0,m}, \mathrm{sign}(m) w_{1,m}, \, \mathrm{sign}(k){\hat{w}}_{2,m}) \\&+\,\mathcal{P}_m(\mathrm{sign}(m){\hat{w}}_{0,m}, \mathrm{sign}(m){\hat{w}}_{1,m}, -\mathrm{sign}(k)w_{2,m}), \end{aligned}$$

where we use the convention that $\mathrm{sign}(0) = 1$.

From the foregoing analysis, in order to show that ${\mathscr {L}}$ is non-negative on $V_1$, it is enough to show that $\mathcal{P}_m (w_0, w_1, w_2) \ge 0$ for any smooth functions $w_0, w_1, w_2 \in C_c^\infty (0,R)$. In addition, it is clear from the definition of $\mathcal{P}_m$ that it suffices to consider $k = 1$ and $m\ge 0$. We consider the cases $m \ge 1$ and $m = 0$ separately.

I. Case $m\ge 1$: Consider first the case $b^4 \ne 3a^2c^2$. In this case, since $u' > 0$, $v' \ne 0$ (see Theorem 1.6) and $u > 0$, we can write $w_0 = v' \eta$, $w_1=u' \xi$, $w_2 = u \zeta$ for $\eta , \xi , \zeta \in C_c^\infty (0,R)$ and use Hardy decomposition trick to obtain

$$\begin{aligned} \mathcal{P}_m (w_0, w_1, w_2)= & {} \int _0^R \Biggl \{ | v'|^2 |\eta '|^2 +(m^2-1)\frac{(v')^2\eta ^2}{r^2}- 2 u v' u' (\xi - \eta )^2 \left[ \frac{b^2}{\sqrt{6}} + c^2 v \right] \\&+\, |u'|^2 |\xi '|^2 + \frac{1}{r^2} \left( | m u' \xi - u \zeta |^2 + | u' \xi - m u \zeta |^2 \right) \\&-\,\frac{2}{r^2} |u'|^2 \xi ^2 + \frac{2}{r^3} uu' \xi ^2+ |u|^2 |\zeta '|^2-\frac{1}{r^2} |u|^2 \zeta ^2 \Biggr \} r\, dr\\= & {} \mathcal {J}_m + \mathcal {I}_m, \end{aligned}$$

where

$$\begin{aligned} \mathcal {J}_m =\int _0^R \Biggl \{ | v'|^2 |\eta '|^2 + |u'|^2 |\xi '|^2 +(m^2-1)\frac{|v'|^2\eta ^2}{r^2} - 2 u v' u' \left[ \frac{b^2}{\sqrt{6}} + c^2 v \right] (\xi -\eta )^2 \Biggr \} r\, dr, \end{aligned}$$

$$\begin{aligned} \mathcal {I}_m = \int _0^R \Biggl \{ \frac{m^2 -1}{r^2} |u'|^2 \xi ^2 + \frac{m^2 }{r^2} |u|^2 \zeta ^2 + |u|^2 |\zeta '|^2 + \frac{2}{r^3} uu' \xi ^2 - \frac{4m}{r^2} u u' \xi \zeta \Biggr \} r\, dr. \end{aligned}$$

Since $v' \big (\frac{b^2}{\sqrt{6}} + c^2 v\big ) < 0$ in (0, R), it is clear that $\mathcal {J}_m \ge 0$. As for $\mathcal {I}_m$, we compute

$$\begin{aligned} \mathcal {I}_m= & {} \int _0^R \Biggl \{\frac{m^2 -1}{r^2} \left( |u'|^2 \xi ^2 + |u|^2 \zeta ^2 \right) - \frac{4(m-1)}{r^2} u u' \xi \zeta \\&+ \frac{1 }{r^2} |u|^2 \zeta ^2 + |u|^2 |\zeta '|^2 + \frac{2}{r^3} uu' \xi ^2 - \frac{4}{r^2} u u' \xi \zeta \Biggr \} r\, dr\\= & {} \int _0^R \Biggl \{ \frac{(m-1)^2}{r^2} \left( |u'|^2 \xi ^2 + |u|^2 \zeta ^2 \right) + \frac{2(m-1)}{r^2} (u' \xi -u \zeta )^2 \\&+ \frac{2u u' }{r^3} \left( \xi - \zeta r\right) ^2 + \frac{|u|^2 }{r^2} \left( \zeta +\zeta ' r \right) ^2\Biggr \} r\, dr \ge 0. \end{aligned}$$

(Here we have used the identity $\int _0^R \big (u^2\zeta \zeta '+uu'\zeta ^2\big )\, dr= \frac{1}{2}\int _0^R (u^2\zeta ^2)'\,dr = 0$.) We conclude that $\mathcal{P}_m (w_0, w_1, w_2) \ge 0$ for $m \ge 1$ and $b^4 \ne 3a^2c^2$.

Let us now turn to the case $b^4 = 3a^2c^2$. By [26, Proposition 3.5], $v \equiv -\frac{s_+}{\sqrt{6}}$ and u is the unique solution of

$$\begin{aligned} u'' + \frac{1}{r}\,u' - \frac{k^2}{r^2}\,u = c^2u\left( u^2 - \frac{s_+^2}{2}\right) , u(0) = 0, u(R) = \frac{s_+}{\sqrt{2}}. \end{aligned}$$

Furthermore $u > 0$ and $u' > 0$.

The argument above for $b^4 \ne 3a^2c^2$ does not apply directly since $v' \equiv 0$ and we cannot write $w_0 = v'\,\eta$ unless $w_0 \equiv 0$. Nevertheless, with the above explicit value of v, the expression for $\mathcal{P}_m$, for $m \ge 0$, simplifies to

$$\begin{aligned} \mathcal{P}_m (w_0, w_1, w_2)= & {} \int _0^R \Biggl \{ | w_0'|^2 + | w_1'|^2 + | w'_2 |^2 + \frac{m^2 }{r^2}|w_0|^2 + \frac{1}{r^2} \left( \big | m w_1 - w_2\big |^2 + \big | w_1 - m w_2\big |^2 \right) \\&+ \frac{1}{2}c^2 (2u^2 + s_+^2) |w_0|^2 + \frac{1}{2}c^2( 6u^2 - s_+^2) |w_1|^2 + \frac{1}{2}c^2 (u^2 - s_+^2) | w_2 |^2 \Biggr \} r\, dr. \end{aligned}$$

It is readily seen that the contribution of $w_0$ is non-negative and uncoupled with $w_1$ and $w_2$. Thus, in proving the positivity of $\mathcal{P}_m$, we can assume without loss of generality that $w_0 \equiv 0$. The foregoing analysis now applies yielding $\mathcal{P}_m(w_0, w_1, w_2) \ge 0$ for all $m \ge 1$.

II. Case $m=0$: Note that

$$\begin{aligned} \mathcal{P}_0(w_0, w_1, w_2)=B(w_1, w_0)+\tilde{F}(w_2) \end{aligned}$$

where B stands for the second variation of ${\mathscr {E}}_R(u, v)$ (see (1.18)) while

$$\begin{aligned} \tilde{F}(w_2)= & {} \int _0^R \bigg \{|w'_2 |^2 +\frac{w_2^2}{r^2}+ \left( -a^2 + \frac{2}{\sqrt{6}}b^2 v+c^2\left( u^2+ v^2\right) \right) | w_2 |^2 \Biggr \} r\, dr\\= & {} \int _0^R (\zeta ')^2u^2 \, rdr \ge 0 \end{aligned}$$

by the computation in the previous case with the Hardy decomposition $w_2=u\zeta$. One concludes that $\mathcal{P}_0(w_0, w_1, w_2) \ge 0$ thanks to (1.19). $\square$

Let us now turn to the study of the kernel of ${\mathscr {L}}[Q]$ in $V_1$.

Proposition 4.3

Under the hypotheses of Proposition 4.1, ${\mathscr {L}}[Q](P) = 0$ for some $P \in V_1$ if and only if the dichotomy in the second part of Theorem 1.4 holds.

Proof

We will consider only the case $k = 1$ and omit the very similar proof for $k = -1$.

Assume that $P \in V_1$ and ${\mathscr {L}}[Q](P) = 0$. Define $P_m$ as in (4.4) so that $P = \sum P_m$. By (4.5) and Proposition 4.1, we have that ${\mathscr {L}}[Q](P_m) = 0$ for all $P_m$.

Define

$$\begin{aligned} Y = \Big \{w: (0,R) \rightarrow {\mathbb {R}}\Big | \sqrt{r}(|w'| + |w|)\in L^2(0,R), \text { and }w(R) = 0 \text { if } R < \infty \Big \}. \end{aligned}$$

(4.6)

For the functionals $\mathcal{P}_m$ defined in the proof of Proposition 4.1, we make the following four claims.

• (i) For all $m \ge 2$ and $w_l \in Y \cap L^2((0,R);\frac{1}{r}\,dr)$, there holds

  $$\begin{aligned} \mathcal{P}_m(w_0, w_1, w_2) \ge \int _0^R \frac{1}{r} \sum _{l = 0}^2 |w_{l}|^2\,dr. \end{aligned}$$

• (ii) For $b^4 \ne 3a^2c^2$ and for all $w_0 \in Y \cap L^2((0,R);\frac{1}{r}\,dr)$ and $w_1, w_2 \in Y$ such that $|w_1 - w_2| \in L^2((0,R);\frac{1}{r}\,dr)$, there holds

  $$\begin{aligned}&\mathcal{P}_1(w_0, w_1, w_2) \\&\quad \ge \int _0^R \bigg \{ \frac{|u|^2 }{r^2} |(r\zeta )'|^2 -2uu'v' \left[ \frac{b^2}{\sqrt{6}} + c^2 v \right] (\xi - \eta )^2 + \frac{2u u' }{r^3} \left( \xi - \zeta r\right) ^2\Biggr \} r\, dr , \end{aligned}$$

  where $\eta = \frac{w_0}{v'}, \xi = \frac{w_1}{u'}$ and $\zeta = \frac{w_2}{u}$.

• (iii) For $b^4 = 3a^2c^2$ and for all $w_0 \in Y \cap L^2((0,R);\frac{1}{r}\,dr)$ and $w_1, w_2 \in Y$ such that $|w_1 - w_2| \in L^2((0,R);\frac{1}{r}\,dr)$, there holds

  $$\begin{aligned} \mathcal{P}_1(w_0, w_1, w_2)&\ge \int _0^R \bigg \{\frac{1}{r^2} |w_0|^2 + \frac{|u|^2 }{r^2} |(r\zeta )'|^2 + \frac{2u u' }{r^3} \left( \xi - \zeta r\right) ^2\Biggr \} r\, dr , \end{aligned}$$

  where $\xi = \frac{w_1}{u'}$ and $\zeta = \frac{w_2}{u}$.

• (iv) For all $w_0, w_2 \in Y \cap L^2((0,R);\frac{1}{r}\,dr)$ and $w_1 \in Y$, there holds

  $$\begin{aligned} \mathcal{P}_0(w_0, w_1, w_2) = B(w_1, w_0) + \int _0^R \frac{(w_2'u - w_2\,u')^2}{u^2}\,r\,dr. \end{aligned}$$

When $w_0, w_1, w_2 \in C_c^\infty (0,R)$, the above claims were established in the proof of Proposition 4.1. They continue to hold in this generality, thanks to Fatou’s lemma, since the left hand sides are quadratic linear forms while the integrands on the right hand sides are non-negative.

Now, we see that $\mathcal{P}_m(w_0, w_1, w_2) = 0$ if and only if one of the following three cases occurs:

• $m \ge 2$ and $w_0 = w_1 = w_2 = 0$,

• or $m = 0$ and $w_2 = 0$ and $B(w_1, w_0) = 0$,

• or $m = 1$ and $(w_0, w_1, w_2) = (tv', tu', \frac{t}{r}u)$ for some constant t.

The conclusion is then readily seen from the above and the fact that

$$\begin{aligned} \frac{\partial Q}{\partial x_1}&=v'(r)\cos \varphi E_0+u'(r)\cos \varphi E_1-k\frac{u(r)}{r}\sin \varphi E_2,\\ \frac{\partial Q}{\partial x_2}&=v'(r)\sin \varphi E_0+u'(r)\sin \varphi E_1+k\frac{u(r)}{r}\cos \varphi E_2. \end{aligned}$$

We omit the details. $\square$

4.3 Stability in the space $V_2$

Proposition 4.4

Let $a^2 \ge 0$, $b^2, c^2 >0$ be fixed constants, $R \in (0, \infty ]$, and $k =\pm 1$. Let (u, v) be a solution of (1.6) on (0, R) under the boundary condition (1.11) such that $u > 0$ and $v < 0$ and let Q be k-radially symmetric solution Q of (1.2) (on $B_R)$ and (1.9) given by (1.4). Then ${\mathscr {L}}[Q](P) > 0$ for all nonzero $P \in V_2$.

Proof

We will consider only the case $k = 1$ and omit the very similar proof for $k = -1$.

Let $P=w_3E_3+w_4 E_4$. Then

$$\begin{aligned} {\mathscr {L}}[Q](P)= & {} \int _0^R \int _0^{2\pi }\bigg \{ \sum _{i=3}^4 \bigg [|\partial _r w_i |^2 \!+\!\frac{1}{r^2} |\partial _\varphi w_i|^2 \!+\! \left( -{a^2} -\frac{b^2}{\sqrt{6}} v + c^2 (u^2+v^2) \right) |w_i|^2 \bigg ]\\&-\frac{b^2 u}{\sqrt{2}}\left( (w_3^2-w_4^2)\cos (k\varphi )+2uw_3w_4\sin (k\varphi ) \right) \bigg \}\, rdrd\varphi . \end{aligned}$$

We will represent

$$\begin{aligned} w= w_3 + i w_4 \end{aligned}$$

to obtain

$$\begin{aligned} {\mathscr {L}}[Q](P) \!=\! \int _0^R \int _0^{2\pi }\bigg \{ |\nabla w |^2 + \left( -{a^2} -\frac{b^2}{\sqrt{6}} v + c^2 (u^2+v^2) \right) |w|^2 -\frac{b^2 u}{\sqrt{2}} \mathrm{Re} (w^2 e^{-i\varphi })\bigg \}\, rdrd\varphi . \end{aligned}$$

Now we can use Fourier decomposition

$$\begin{aligned} w = \sum _{n \in {\mathbb {Z}}} \xi _n (r) e^{i n\varphi }. \end{aligned}$$

We note that $\xi _n \in Y \cap L^2((0,R);\frac{1}{r}dr)$ for $n \ne 0$ and $\xi _0 \in Y$, where Y is defined by (4.6).

It is clear that

$$\begin{aligned} \int _0^{2\pi } w^2 e^{-i \varphi } \, d\varphi = \int _0^{2\pi } \sum _{n,m \in {\mathbb {Z}}} \xi _n (r) \xi _m (r) e^{i (n+m-1)\varphi } \, d\varphi = 4\pi \sum _{n=1}^\infty \xi _n (r) \xi _{1-n} (r). \end{aligned}$$

Therefore

$$\begin{aligned} \frac{{\mathscr {L}}[Q](P)}{2 \pi } \!= & {} \! \sum _{n=1}^\infty \int _0^R \left[ |\xi _n'|^2 + |\xi _{1-n}'|^2 + \frac{n^2}{r^2} |\xi _n |^2 + \frac{(1-n)^2}{r^2} |\xi _{1-n} |^2 \right. \\&+\; \left. \left( -{a^2} -\frac{b^2}{\sqrt{6}} v \!+\! c^2 (u^2+v^2) \right) (|\xi _n|^2 \!+\! |\xi _{1-n}|^2) - \sqrt{2} b^2 u \mathrm{Re} (\xi _n \xi _{1-n} ) \right] r\, dr\\= & {} J_1+J_2, \end{aligned}$$

where $J_1$ and $J_2$ correspond to $n=1$ and $n \ge 2$.

Estimating $J_2$. We use Hardy decomposition trick $\xi _n = u \eta _n$ for $n\ge 2$ and $n \le -1$ to obtain

$$\begin{aligned} J_2= & {} \sum _{n=2}^\infty \int _0^R \left[ |\xi _n'|^2 + |\xi _{1-n}'|^2 + \frac{n^2}{r^2} |\xi _n |^2 + \frac{(1-n)^2}{r^2} |\xi _{1-n} |^2 \right. \\&+\; \left. \left( -{a^2} -\frac{b^2}{\sqrt{6}} v + c^2 (u^2+v^2) \right) (|\xi _n|^2 + |\xi _{1-n}|^2) - \sqrt{2} b^2 u \mathrm{Re} (\xi _n \xi _{1-n} ) \right] r\, dr\\= & {} \sum _{n=2}^\infty \int _0^R \left[ |\eta _n'|^2 + |\eta _{1-n}'|^2 + \frac{n^2-1}{r^2} |\eta _n |^2 + \frac{(1-n)^2-1}{r^2} |\eta _{1-n} |^2 \right. \\&\left. -\frac{3 b^2}{\sqrt{6}} v (|\eta _n|^2 + |\eta _{1-n}|^2) - \sqrt{2} b^2 u \mathrm{Re} (\eta _n \eta _{1-n} ) \right] u^2 r\, dr\\\ge & {} \sum _{n=2}^\infty \int _0^R \frac{b^2}{\sqrt{2}}(-\sqrt{3} v - u)(|\eta _n|^2 + |\eta _{1-n}|^2) u^2 r\, dr\\\ge & {} \sum _{n=2}^\infty \int _0^R \frac{b^2}{\sqrt{2}}(-\sqrt{3} v - u)(|\xi _n|^2 + |\xi _{1-n}|^2) r\, dr. \end{aligned}$$

Using the fact that $-\sqrt{3} v -u > 0$ in (0, R) (cf. (2.2)), we obtain that $J_2>0$ for nonzero modes $\{\xi _n\}_{n\ne 0,1}$. (Strictly speaking the above estimates are first shown for $\xi _n \in C_c^\infty (0,R)$ and then extend to $\xi _n \in Y \cap L^2((0,R);\frac{1}{r}\,dr)$ by density.)

Estimating $J_1$. For $\xi _0, \xi _1 \in C_c^\infty (0,R)$, we have by Hardy decomposition trick for $\xi _0 = v \eta _0$ and $\xi _1 = u \eta _1$:

$$\begin{aligned} J_1= & {} \int _0^R \left[ |\xi _1'|^2 + |\xi _{0}'|^2 + \frac{1}{r^2} |\xi _1 |^2 \right. \\&+\; \left. \left( -{a^2} -\frac{b^2}{\sqrt{6}} v + c^2 (u^2+v^2) \right) (|\xi _1|^2 + |\xi _{0}|^2) - \sqrt{2} b^2 u \mathrm{Re} (\xi _1 \xi _{0} ) \right] r\, dr.\\= & {} \int _0^R \left[ |\eta _1'|^2 u^2 + |\eta _0'|^2 v^2 - \frac{3 b^2}{\sqrt{6}} v |\xi _1|^2 - \frac{b^2}{\sqrt{6} v} u^2 |\xi _0|^2 - \sqrt{2} b^2 u \mathrm{Re} (\xi _1 \xi _0 ) \right] r\, dr\\= & {} \int _0^R \left[ |\eta _1'|^2 u^2 + |\eta _0'|^2 v^2 - \frac{b^2}{v \sqrt{6}} \left| \sqrt{3} v \xi _1 + u \xi _0 \right| ^2 \right] r\, dr. \end{aligned}$$

As in the proof of Proposition 4.3, this leads to³

$$\begin{aligned} J_1 \ge \int _0^R \left[ |\eta _1'|^2 u^2 + |\eta _0'|^2 v^2 - \frac{b^2}{v \sqrt{6}} \left| \sqrt{3} v \xi _1 + u \xi _0 \right| ^2 \right] r\, dr \end{aligned}$$

for $\xi _0 \in Y$ and $\xi _1 \in Y \cap L^2((0,R);\frac{1}{r}\,dr)$. Therefore, $J_1>0$ for nonzero modes $\{\xi _n\}_{n=0,1}$. We conclude that ${\mathscr {L}}[Q](P) >0$. $\square$

4.4 Proof of Theorem 1.4

The theorem is a consequence of Propositions 4.1, 4.3 and 4.4.$\Box$

Appendix

Lemma 5.1

Assume that $a^2 \ge 0, b^2, c^2 > 0$. Let

$$\begin{aligned} f(x,y) = -\frac{a^2}{2}(x^2 + y^2) + \frac{c^2}{4}(x^2 + y^2)^2 - \frac{b^2}{3\sqrt{6}}y(y^2 - 3x^2). \end{aligned}$$

Then

$$\begin{aligned} \min _{{\mathbb {R}}^2} f = -\frac{a^2}{3} s_+^2 - \frac{2b^2}{27}\,s_+^3 + \frac{c^2}{6} s_+^4 \end{aligned}$$

which is attained at (and only at) $\big (0,\frac{2}{\sqrt{6}}s_+\big )$ and $\big (\pm \frac{1}{\sqrt{2}}s_+,- \frac{1}{\sqrt{6}}s_+\big )$. Furthermore, the Hessian of f at all these critical points is positive definite.

Proof

We write $x = r\,\sin \varphi$ and $y = r\cos \varphi$ for some $r \ge 0$ and $\varphi \in [0,2\pi )$. Then

$$\begin{aligned} f(x,y) = -\frac{a^2}{2}r^2 + \frac{c^2}{4}r^4 - \frac{b^2}{3\sqrt{6}}\,r^3 \cos 3\varphi \ge -\frac{a^2}{2}r^2 + \frac{c^2}{4}r^4 - \frac{b^2}{3\sqrt{6}}\,r^3 =: {\tilde{f}}(r). \end{aligned}$$

It is easy to check that ${\tilde{f}}$ has three critical points, $r = 0$ and $r = \frac{2}{\sqrt{6}}s_\pm$ where the first one is a local maximum point and the other two are local minimum points. The global minimum of ${\tilde{f}}$ is then verified to be achieved at $r = \frac{2}{\sqrt{6}}s_+$. We have thus shown that

$$\begin{aligned} f(x,y) \ge {\tilde{f}}\left( \frac{2}{\sqrt{6}}s_+\right) = -\frac{a^2}{3} s_+^2 - \frac{2b^2}{27}\,s_+^3 + \frac{c^2}{6} s_+^4 , \end{aligned}$$

and equality is attained if and only if $r =\frac{2}{\sqrt{6}}s_+$ and $\varphi \in \{0, \frac{2\pi }{3}, \frac{4\pi }{3}\}$. The first assertion follows.

Now a computation using $-a^2 - \frac{b^2}{3} s_+ + \frac{2}{3}c^2\,s_+^2 = 0$ leads to

$$\begin{aligned} D^2 f\left( 0,\frac{2}{\sqrt{6}}s_+\right)&= \left[ \begin{array}{cc} b^2\,s_+ &{} 0\\ 0&{} \frac{1}{3}(3a^2 + b^2\,s_+) \end{array}\right] ,\\ D^2 f\left( \pm \frac{1}{\sqrt{2}}s_+,- \frac{1}{\sqrt{6}}s_+\right)&= \left[ \begin{array}{cc} c^2 s_+^2 &{} \pm \frac{1}{\sqrt{3}}(-c^2\,s_+^2+ b^2\,s_+)\\ \pm \frac{1}{\sqrt{3}}(-c^2\,s_+^2+ b^2\,s_+) &{} \frac{1}{3}(c^2\,s_+^2 + 2b^2\,s_+) \end{array}\right] , \end{aligned}$$

from which the last assertion follows. $\square$