1 Introduction

If a black hole solution is known, it is very important to carry out the stability analysis of the black hole. At the early stage of studying the Schwarzschild black hole, a conventional method to determine the stability is to solve the linearized Einstein equation by choosing even-and odd-parity perturbations under the Regge–Wheeler gauge for graviton, which leads to two Schrödinger equations: Regge–Wheeler equation [1] and Zerilli equation [2]. One may conclude that the Schwarzschild black hole is stable because their potentials are positive definite for the whole region outside the black hole, implying that there is no exponentially growing modes [3, 4]. Equivalently, the stability of a black hole depends on the sign of the imaginary part \(\omega _\mathrm{I}\) of their quasinormal frequencies \(\omega =\omega _\mathrm{R}+i \omega _\mathrm{I}\) when considering the time dependence of \(e^{-i\omega t}\) [5]. If \(\omega _\mathrm{I}\) is negative, the black hole is stable. The real part \(\omega _\mathrm{R}\) has no bearing on stability properties. Furthermore, the unstable condition of a black hole was suggested to be \(\omega _\mathrm{R}=0\) and \(\omega _\mathrm{I}\ge 0\) [6].

On the other hand, the stability analysis of the Schwarzschild black hole obtained from higher derivative gravity is not an easy task because it contains the second-order equation for a massive graviton. A conventional stability method designed for a graviton with two degrees of freedom (2 DOF) is not suitable for studying the massive graviton (5 DOF) [7] which is propagating on the black hole and de Sitter spacetimes [8]. However, if one considers a lower dimensional massive gravity, the situation is not so complicated. Reminding that the three dimensional Einstein gravity is a gauge theory, any propagating spin-2 mode belongs to massive graviton which can be obtained from three-dimensional massive gravity theories. Further, these theories are classified into parity-even and parity-odd theories.

Recently, it was shown that the BTZ black hole [9, 10] is stable for all \(\mu \) (Chern-Simons coupling constant) against the massive spin-2 perturbations in the topologically massive gravity (TMG [11], parity-odd theory) by demanding boundedness of the perturbation at the horizon [12]. On the other hand, it was suggested that the BTZ black hole is stable for \(m^2>1/2\ell ^2\) in new massive gravity (NMG, parity-even theory) [13] by computing quasinormal frequencies and performing the \(s\)-mode analysis [14].

Because of different parity, one uses different stability analysis for the BTZ black hole. Solving the first-order differential tensor equation algebraically together with the boundary conditions, we obtain all quasinormal frequencies of massive graviton for parity-odd theories [15]: TMG and generalized massive gravity (GMG) [16]. If \(\omega _\mathrm{I} < 0\), the black hole seems to be stable against the massive graviton perturbation.

Given the parity-odd first order linearized equation, one obtains its second-order linearized equation after squaring it, which belongs to the parity-even theory, giving the ambiguity on sign of the mass. Because of this ambiguity, someone prefers solving the first-order equation directly [15], instead of the second-order equation. Off-critical point, the parity-even gravity theory usually provides the second-order linearized equation after choosing the transverse-traceless gauge. It is known that “solving directly the second-order massive equation” is a formidable task for the BTZ black hole spacetimes. Fortunately, after choosing the BTZ black string perturbation for massive graviton [17], the \(s\)-mode analysis may be performed using two Schrödinger equations with frequency-dependent potentials [14].

In this work, we study the massive graviton stability of the BTZ black hole obtained from three-dimensionalFootnote 1 massive gravities which are classified into the parity-odd theories (TMG, GMG) and parity-even gravity theories (NMG, six-derivative gravity (SDG) [19]). For the second-order Schrödinger equation with an effective potential, we analyze the massive graviton stability by checking if the potential is positive for the whole range outside an event horizon. However, we should extend the stability condition (2.26) for asymptotically flat spacetimes to (2.29) for asymptotically AdS spacetimes. In this case, the stability condition of \(s\)-mode is given by the generalized Breitenlohner-Freedman bound for spin-2 field. On the other hand, for a first-order massive graviton equation, we perform the stability analysis by using the quasinormal frequencies \((\omega =\omega _\mathrm{R}+i \omega _\mathrm{I})\). If \(\omega _\mathrm{I}\) is negative when considering the time dependence of \(e^{-i\omega t}\) [5], the black hole is stable. Finally, we explore the black hole stability connection between the parity-odd and parity-even (\(s\)-mode) massive gravity theories.

2 Parity-even massive gravities

In this work, we consider the (non-rotating) BTZ black hole solution [9] which is a solution to all massive gravity theories. Its line element is given by

$$\begin{aligned} ds_{\mathrm{BTZ}}^2&= \bar{g}_{\mu \nu }dx^{\mu }dx^{\nu }\nonumber \\&= -\left( -\mathcal{M}+\frac{r^2}{\ell ^2}\right) dt^2+ \left( -\mathcal{M}+\frac{r^2}{\ell ^2}\right) ^{-1}dr^2+r^2d\phi ^2, \end{aligned}$$
(2.1)

where \(\mathcal{M}\) is the ADM mass given to be \(\mathcal{M}=r_+^2/\ell ^2\) with the horizon radius \(r_+\) and AdS\(_3\) curvature radius \(\ell \). Throughout the paper, the overbar denotes the background metric (2.1) for the BTZ black hole. The Ricci scalar, Ricci tensor, Riemann tensor can be written in terms of the background metric (2.1) as follows:

$$\begin{aligned} \bar{R}=6\Lambda ,\quad \bar{R}_{\mu \nu }=2\Lambda \bar{g}_{\mu \nu },\quad \bar{R}_{\mu \nu \rho \sigma }=\Lambda (\bar{g}_{\mu \rho }\bar{g}_{\nu \sigma } -\bar{g}_{\mu \sigma }\bar{g}_{\nu \rho }), \end{aligned}$$
(2.2)

where \(\Lambda =-1/\ell ^2\). Also we adopt a notation of \((-,+,+)\) and unit of \(2\kappa ^2=1\). For the perturbation around the BTZ black hole

$$\begin{aligned} g_{\mu \nu }=\bar{g}_{\mu \nu }+h_{\mu \nu }, \end{aligned}$$
(2.3)

the linearized Ricci tensor and Ricci scalar are given by

$$\begin{aligned} \delta R_{\mu \nu }(h)&= \frac{1}{2}\Big (\bar{\nabla }_{\mu } \bar{\nabla }^{\rho }h_{\rho \nu }+\bar{\nabla }_{\nu } \bar{\nabla }^{\rho }h_{\rho \mu }-\bar{\nabla }^2h_{\mu \nu } -\bar{\nabla }_{\mu }\bar{\nabla }_{\nu }h\Big )+3\Lambda h_{\mu \nu }-\Lambda \bar{g}_{\mu \nu }h,\nonumber \\ \end{aligned}$$
(2.4)
$$\begin{aligned} \delta R(h)&= \bar{\nabla }_{\alpha }\bar{\nabla }_{\beta } h^{\alpha \beta }-\bar{\nabla }^2h-2\Lambda h. \end{aligned}$$
(2.5)

Let us first introduce a three-dimensional massive gravity proposed by Fierz and Pauli (FP) [7] whose action is given by

$$\begin{aligned} S_\mathrm{FP}=S_\mathrm{bl}(h)-\frac{M_\mathrm{FP}^2}{4}\int d^3x\sqrt{-g}\left( h_{\mu \nu }h^{\mu \nu }-h^2\right) , \end{aligned}$$
(2.6)

where \(M_\mathrm{FP}^2\) is a mass parameter and \(S_\mathrm{bl}(h)\) is the bilinear form of the Einstein-Hilbert action with a cosmological constant \(\Lambda \). It is well known that the linearized Einstein equation can be written as [8]

$$\begin{aligned} \Big (\bar{\nabla }^{2}-2\Lambda -M_\mathrm{FP}^2\Big )h_{\mu \nu }^\mathrm{FP}=0, \end{aligned}$$
(2.7)

which is considered as a simplest equation for a massive graviton with 2 DOF on the BTZ black hole spacetimes. In deriving this equation, we have used the consistency condition of the linearized Bianchi identity

$$\begin{aligned} \bar{\nabla }_\mu h^{\mu \nu }=0,~~h^{\mu }_{~\mu }=0, \end{aligned}$$
(2.8)

which is considered as the transverse-traceless (TT) gauge.Footnote 2

Now we consider the parity-even six-derivative gravity (SDG) [19], whose action is given by

$$\begin{aligned} S_{\mathrm{SDG}}&= \int d^3x\sqrt{-g} \Big [\sigma R-2\lambda _S+\alpha R^2+\beta R_{\mu \nu }R^{\mu \nu } +a_1\nabla _{\mu }R\nabla ^{\mu }R\nonumber \\&+a_2\nabla _{\rho }R_{\mu \nu }\nabla ^{\rho }R^{\mu \nu }\Big ], \end{aligned}$$
(2.9)

where \(\sigma =0,\pm 1\) is a dimensionless parameter, \(\lambda _S\) is a cosmological parameter with mass dimension 2. Here parameters \(\alpha (\beta )\) have mass dimension \(-2\) and \(a_1(a_2)\) have \(-4\). We remark that when choosing \(a_1=a_2=0, \sigma =1,\) and \(8\alpha +3\beta =0\), the action (2.9) reduces to the NMG action [13] as

$$\begin{aligned} S_\mathrm{NMG}=\int d^3x\sqrt{-g}\Bigg [R-2\lambda _S-\frac{1}{m^2}\left( R_{\mu \nu }R^{\mu \nu }-\frac{3}{8}R^2\right) \Bigg ], \end{aligned}$$
(2.10)

where \(m^2\) is a mass parameter with dimension \(2\).

Varying the action (2.9) with respect to \(g^{\mu \nu }\) leads to the equation

$$\begin{aligned} \sigma \Big (R_{\mu \nu }-\frac{1}{2}g_{\mu \nu }R\Big )+\lambda _S g_{\mu \nu }+E_{\mu \nu }+H_{\mu \nu }=0 \end{aligned}$$
(2.11)

with

$$\begin{aligned} E_{\mu \nu }\!&= \!\beta \Bigg [\!-\!\frac{1}{2}g_{\mu \nu }R_{\rho \sigma }R^{\rho \sigma } \!+\!2R_{\mu \rho \nu \sigma }R^{\rho \sigma }\!+\!\nabla _{\gamma }\nabla ^{\gamma } R_{\mu \nu }\!+\!\frac{1}{2}g_{\mu \nu }\nabla _{\gamma }\nabla ^{\gamma }R \!-\!\nabla _{\mu }\nabla _{\nu }R\Bigg ]\nonumber \\&+\,\alpha \Big [2RG_{\mu \nu }\!+\!2g_{\mu \nu }\nabla _{\gamma }\nabla ^{\gamma }R \!-\!2\nabla _{\mu }\nabla _{\nu }R\Big ], \end{aligned}$$
(2.12)
$$\begin{aligned} H_{\mu \nu }\!&= \!a_1\Bigg [\nabla _{\mu }R\nabla _{\nu }R\!-\!2R_{\mu \nu }\nabla ^2R \!-\!\frac{1}{2}g_{\mu \nu }\nabla _{\rho }R\nabla ^{\rho }R\!-\!2(g_{\mu \nu }\nabla ^2 \!-\!\nabla _{\mu }\nabla _{\nu }\nabla ^2)R\Bigg ]\nonumber \\&a_2\Bigg [\nabla _{\mu }R_{\rho \sigma }\nabla _{\nu }R^{\rho \sigma } \!-\!\frac{1}{2}g_{\mu \nu }\nabla _{\gamma }R_{\rho \sigma }\nabla ^{\gamma }R^{\rho \sigma } \!-\!\nabla ^2R_{\mu \nu }-g_{\mu \nu }\nabla ^{\rho }\nabla ^{\sigma }\nabla ^2R_{\rho \sigma } \nonumber \\&+2\nabla ^{\rho }\nabla _{(\mu }\nabla ^2R_{\nu )\rho } \!+\!2\nabla ^{\rho }R_{\rho \sigma }\nabla _{(\mu }R_{\nu )}^{\sigma } \!+\!2R_{\rho \sigma }\nabla ^{\rho }\nabla _{(\mu }R_{\nu )}^{\sigma } \!-\!2R_{\sigma (\mu }\nabla ^2R_{\nu )}^{\sigma }\nonumber \\&-2\nabla _{\rho }R_{\sigma (\mu }\nabla _{\nu )}R^{\rho \sigma } \!-\!2R_{\sigma (\mu }\nabla ^{\rho }\nabla _{\nu )}R_{\rho }^{\sigma }\Bigg ]. \end{aligned}$$
(2.13)

We note that the BTZ black hole solution (2.1) to Eq. (2.11) is allowed only when choosing \(\lambda _{S}=\sigma \Lambda -2(3\alpha +\beta )\Lambda ^2\). Taking into account the perturbation (2.3) and plugging the TT gauge (2.8) into Eq. (2.11), we obtain the sixth-order differential perturbation equation, which can be factored into three piecesFootnote 3:

$$\begin{aligned} \Big [\bar{\nabla }^2-2\Lambda \Big ] \Big [\bar{\nabla }^2-2\Lambda -M_+^2\Big ] \Big [\bar{\nabla }^2-2\Lambda -M_-^2\Big ]h_{\mu \nu }=0. \end{aligned}$$
(2.14)

Here the mass parameters \(M^2_{\pm }\) denote

$$\begin{aligned} M_{\pm }^2=\frac{\beta }{2a_2}-\Lambda \pm \frac{1}{2a_2} \sqrt{10a_2^2\Lambda ^2-6a_2\beta \Lambda +4a_2\sigma +\beta ^2}. \end{aligned}$$
(2.15)

In Eq. (2.14), we read off two massive equations

$$\begin{aligned} \Big [\bar{\nabla }^2-2\Lambda -M_+^2\Big ]h_{\mu \nu }^{M_+}=0,~~~ \Big [\bar{\nabla }^2-2\Lambda -M_-^2\Big ]h_{\mu \nu }^{M_-}=0 \end{aligned}$$
(2.16)

off-critical points (\(M^2_+\not =M^2_-\)). They describe 4 DOF for two massive gravitons.

Also, we note that for the NMG (2.10), the linearized equation is given by

$$\begin{aligned} \Big [\bar{\nabla }^2-2\Lambda -M^2_\mathrm{NMG}\Big ]h_{\mu \nu }^\mathrm{NMG}=0, ~~M_\mathrm{NMG}^2~=~m^2-\frac{1}{2\ell ^2} \end{aligned}$$
(2.17)

off critical point (\(m^2\ne 1/2\ell ^2\)) and off-decoupling limitFootnote 4 (\(m^2\ne 0\)), which describes 2 DOF for a massive graviton in three dimensional spacetimes.

2.1 \(s\)-mode stability analysis

We are now in a position to perform the stability analysis of massive gravitons satisfying Eqs. (2.7), (2.16), and (2.17). We propose that they are propagating on the BTZ black hole background (2.1). For this purpose, inspired by the BTZ black string perturbations [17], we consider the following two distinct (orthogonal) perturbations ansatz [14]: the type I has two off-diagonal components \(h_0\) and \(h_1\)

$$\begin{aligned} h^I_{\mu \nu }=\left( \begin{array}{c@{\quad }c@{\quad }c} 0 &{} 0 &{} h_0(r) \\ 0 &{} 0 &{} h_1(r) \\ h_0(r) &{} h_1(r) &{} 0 \end{array} \right) e^{-i\omega t}e^{ik\phi } \,, \end{aligned}$$
(2.18)

while for the type II, the metric tensor takes the form with four components \(H_0,~H_1,~H_2,\) and \(H_3\) as

$$\begin{aligned} h^{II}_{\mu \nu }=\left( \begin{array}{c@{\quad }c@{\quad }c} H_0(r) &{} H_1(r) &{} 0 \\ H_1(r) &{} H_2(r) &{} 0 \\ 0 &{} 0 &{} H_3(r) \end{array} \right) e^{-i\omega t}e^{ik\phi } \,. \end{aligned}$$
(2.19)

In this work, we focus on \(s\)-mode (\(k=0\)) case for simplicity. Importantly, Eqs. (2.7), (2.17), and (2.16) can be combined into a single massive equation

$$\begin{aligned} \left( \bar{\nabla }^2-2\Lambda -M_{i}^2\right) h_{\mu \nu }^{M_i}=0, \end{aligned}$$
(2.20)

where \(M_i^2(i=1,2,3,4)\) is given by

$$\begin{aligned} M_1^2=M_{\mathrm{FP}}^2,~ M_2^2=M_\mathrm{NMG}^2,~M_3^2=M_{+}^2,~M_4^2=M_{-}^2. \end{aligned}$$
(2.21)

For type I, plugging (2.18) into (2.20) and eliminating \(h_1(r)\) from \((t,\phi )\) and \((r,\phi )\) components of (2.20) lead to the Schrödinger equation as

$$\begin{aligned} \frac{d^2\Phi _{i}}{dr^{*2}}+\Big [\omega ^2-V^\mathrm{I}_{\Phi _i}\Big ]\Phi _{i}=0, \end{aligned}$$
(2.22)

where \(r^*\) is the tortoise coordinate defined by the relation of \(dr^*=\ell ^2dr/(r^2-r_+^2)\). Here, a new field \(\Phi _i\) is defined by \(\Phi _i=h_0/\sqrt{r\{M_i^2(r^2-r_+^2)/\ell ^2-\omega ^2\}}\), and \(V^\mathrm{I}_{\Phi _i}\) is the \(\omega \)-dependent potential given by

$$\begin{aligned} V^\mathrm{I}_{\Phi _i}(\omega ,r)&= \frac{r^2-r_+^2}{\ell ^2} \left[ M_{i}^2+\frac{15}{4\ell ^2}-\frac{3r_+^2}{4\ell ^2r^2}+ \frac{3M_i^4 r^2(r^2-r_+^2)}{\ell ^6\left\{ M_i^2(r^2-r_+^2)/ \ell ^2-\omega ^2\right\} ^2}\right. \nonumber \\&\left. +\frac{2M_i^2(2r_+^2-3r^2)}{\ell ^4\left\{ M_i^2(r^2-r_+^2)/ \ell ^2-\omega ^2\right\} }\right] . \end{aligned}$$
(2.23)

We show that for \(M_i^2\ge 0\), all potentials \(V^\mathrm{I}_{\Phi _i}\) are always positive definite for the whole range of \(r_+\le r\le \infty \). This may imply that for \(M_i^2\ge 0\), the BTZ black hole is stable against type I perturbation.

On the other hand, in type II case, substituting (2.19) into (2.20) and after some manipulations, \((t,\phi )\) component of (2.20) can be written as the other Schrödinger equation:

$$\begin{aligned} \frac{d^2\Psi _i}{dr^{*2}}+[\omega ^2-V^\mathrm{II}_{\Psi _i}(\omega ,r)]\Psi _i=0, \end{aligned}$$
(2.24)

where \(\Psi _i=H_1\sqrt{r(r^2-r_+^2)^2}/ \sqrt{M_i^2(r_+^2-r^2)\ell ^2+(2r_+^2-r^2)+\omega ^2\ell ^4}\) and the \(\omega \)-dependent potential \(V^\mathrm{II}_{\Psi _i}\) is given by

$$\begin{aligned} V^\mathrm{II}_{\Psi _i}(\omega ,r)\!&= \!\frac{r^2\!-\!r_+^2}{\ell ^2} \left[ M_i^2\!+\!\frac{7}{4\ell ^2}\!-\!\frac{3r_+^2}{4\ell ^2r^2} \!+\!\frac{3r^2(M_i^2\!+\!1/\ell ^2)^2(r^2\!-\!r_+^2)}{\ell ^6 \left\{ M_i^2(r_+^2\!-\!r^2)/\ell ^2\!+\!(2r_+^2\!-\!r^2)/\ell ^4\!+\!\omega ^2\right\} ^2}\right. \nonumber \\&\left. +\frac{4(M_i^2\!+\!1/\ell ^2)(r^2\!-\!r_+^2)}{\ell ^4\left\{ M_i^2(r_+^2\!-\!r^2)/\ell ^2\!+\!(2r_+^2\!-\!r^2)/\ell ^4\!+\!\omega ^2\right\} }\right] . \end{aligned}$$
(2.25)

We note that for \(M_i^2\ge 0\), all potential \(V^\mathrm{II}_{\Psi _i}\) is always positive definite for the whole range of \(r_+\le r\le \infty \), which states that the BTZ black hole is stable against type-II perturbation.

Hence, if one applies type I and II perturbations to the parity-even massive gravities, the stability conditions of the BTZ black hole seem to be

$$\begin{aligned} M_\mathrm{FP}^2 \ge 0,~~~~ m^2\ge \frac{1}{2\ell ^2}, ~~~~~ M_{\pm }^2\ge 0 \end{aligned}$$
(2.26)

in FP, NMG, and SDG, respectively. However, these conditions are suitable for asymptotically flat spacetimes. We remind the reader that our spacetime is asymptotically anti de Sitter spacetimes. Therefore, we have to point out what is the stability condition of a massive graviton propagating on the AdS\(_3\) spacetimes. To see this explicitly, let us consider asymptotically AdS\(_3\) spacetimes, which corresponds to a large \(r\) limit (\(r^{*}\rightarrow 0\)) in Eq. (2.1). In this limit, the potentials (2.23) and (2.25) take the same form when expressing them in terms of a tortoise coordinate \(r^*\)

$$\begin{aligned} V^\mathrm{I}_{\Phi _i},~V^\mathrm{II}_{\Psi _i}\sim \frac{\xi }{r^{*2}}, \end{aligned}$$
(2.27)

where

$$\begin{aligned} \xi =\ell ^2\left( M_i^2+\frac{3}{4\ell ^2}\right) . \end{aligned}$$
(2.28)

As \(r^{*}\) approaches \(0\), Eqs. (2.22) and (2.24) become one-dimensional Schrödinger equation with an inverse square potential of the strength \(\xi \) and the energy \(E=\omega ^2\). It is known [21, 22] that in this case, if \(\xi \) satisfies the condition,

$$\begin{aligned} \xi \ge -\frac{1}{4}~~\Rightarrow ~~M_i^2\ge -\frac{1}{\ell ^2}, \end{aligned}$$
(2.29)

the energy spectrum is always continuous and positive. It is worth noting that the stability condition (2.29) is consistent with the regularized condition at \(r^{*}=0\). Importantly, the stability condition (2.29) is exactly the same with the Breitenlohner-Freedman (BF) bound [23] for a massive spin-2 field in AdS\(_3\) spacetimes [24, 25]

$$\begin{aligned} \Big [\nabla _{(\mathrm AdS)}^2-2\Lambda -M_{(\mathrm AdS)}^2\Big ]h_{\mu \nu }=0~~~\Rightarrow ~~~M_{(\mathrm AdS)}^2\ge M_\mathrm{BF}^2=-\frac{1}{\ell ^2}. \end{aligned}$$
(2.30)

Hence, we should extend the stability condition (2.26) for asymptotically flat spacetimes to the stability condition (2.29) for asymptotically AdS spacetimes.

Finally, we dictates the stability condition of the BTZ black hole

$$\begin{aligned} M_\mathrm{FP}^2\ge -\frac{1}{\ell ^2},~~~~ m^2\ge -\frac{1}{2\ell ^2}, ~~~~\mathrm{and}~~ M_{\pm }^2\ge -\frac{1}{\ell ^2} \end{aligned}$$
(2.31)

off-critical points (\(m^2\ne 1/2\ell ^2,~M_{\pm }^2\ne 0\)) and off-decoupling limit (\(m^2\ne 0\)), when using the \(s\)-mode analysis for the parity-even massive gravity theories.

3 Parity-odd massive gravities

A parity-odd massive gravity in three dimensions was first introduced by Deser, Jackiw, and Templeton [18]. The TMG action includes a gravitational Chern-Simons term, which reveals parity-violation or ‘odd’ parity. In this section, we introduce two parity-odd massive gravities of TMG and GMG, and investigate the stability analysis of the BTZ black hole in those gravities.

3.1 TMG

The action of TMG with a negative cosmological constant is given by [18]

$$\begin{aligned} S_{\mathrm{TMG}}=\int d^3x\sqrt{-g}\left[ R-2\Lambda +\frac{1}{2\mu } \epsilon ^{\lambda \mu \nu }\Gamma _{\lambda \sigma }^{\rho } \left( \partial _{\mu }\Gamma _{\rho \nu }^{\sigma } +\frac{2}{3}\Gamma _{\mu \tau }^{\sigma }\Gamma _{\nu \rho }^{\tau }\right) \right] \!,\quad \quad \end{aligned}$$
(3.1)

where \(\mu \) is a parameter with mass dimension 1. The Einstein equation takes the form

$$\begin{aligned} R_{\mu \nu }-\frac{1}{2}g_{\mu \nu }R+\Lambda g_{\mu \nu } +\frac{1}{\mu }C_{\mu \nu }=0, \end{aligned}$$
(3.2)

where the Cotton tensor \(C_{\mu \nu }\) is defined by

$$\begin{aligned} C_{\mu \nu }\equiv \epsilon _{\mu }^{~\alpha \beta }\nabla _{\alpha } \left( R_{\beta \nu }-\frac{1}{4}g_{\beta \nu }R\right) . \end{aligned}$$
(3.3)

Introducing the perturbation (2.1) and applying the TT gauge condition (2.8) to the linearized equation of (3.2), we arrive at

$$\begin{aligned} \Big [\bar{\nabla }^2-2\Lambda \Big ]\Big [h_{\mu \nu } +\frac{1}{\mu } \epsilon _{\mu }^{~\alpha \beta }\bar{\nabla }_{\alpha }h_{\beta \nu }\Big ]=0. \end{aligned}$$
(3.4)

From (3.4), we read off the first-order differential equation for a massive graviton

$$\begin{aligned} \epsilon _{\mu }^{~\alpha \beta }\bar{\nabla }_{\alpha }h_{\beta \nu } +\mu h_{\mu \nu } =0. \end{aligned}$$
(3.5)

One can easily check that squaring it [equivalently, by applying the first-order operator \(\epsilon _{\sigma }^{~\rho \mu }\bar{\nabla }_\rho -\mu \delta _{\sigma }^{\mu }\) to (3.5)] leads to the second-order equation

$$\begin{aligned} \Big [\bar{\nabla }^2-2\Lambda -M_\mathrm{TMG}^2\Big ]h_{\sigma \nu }=0 \end{aligned}$$
(3.6)

with \(M_\mathrm{TMG}^2=\mu ^2-1/\ell ^2\). Using the bound given by the stable condition (2.31), we have

$$\begin{aligned} M_\mathrm{TMG}^2 \ge -\frac{1}{\ell ^2} \rightarrow \mu ^2 \ge 0 \rightarrow |\mu | \ge 0 \end{aligned}$$
(3.7)

which is consistent with that obtained in [12], indicating that the BTZ black hole is stable for all \(\mu \) against the massive spin-2 perturbation in TMG by demanding boundedness of the perturbation at the horizon. The latter condition eliminates modes which are growing in time and obeying the generalized boundary conditions at asymptotic infinity. At this stage, we emphasize that the authors in [12] have used not the first-order Eq. (3.5) itself but a second-order hypergeometric equation obtained by transforming two first-order equations when analyzing the stability of the BTZ black hole.

3.2 GMG

We consider the GMG action which consists of NMG and gravitational Chern-Simons term as [13, 26]

$$\begin{aligned} S_{\mathrm{GMG}}&= \int d^3x\sqrt{-g}\left[ \sigma R-2\lambda _G +\frac{1}{m^2}\left( R_{\mu \nu }R^{\mu \nu }-\frac{3}{8}R^2\right) \right. \nonumber \\&\left. +\frac{1}{2\mu } \epsilon ^{\lambda \mu \nu }\Gamma _{\lambda \sigma } ^{\rho }\Big (\partial _{\mu }\Gamma _{\rho \nu }^{\sigma }+\frac{2}{3} \Gamma _{\mu \tau }^{\sigma }\Gamma _{\nu \rho }^{\tau } \Big )\right] , \nonumber \\ \end{aligned}$$
(3.8)

where \(\lambda _G\) is a cosmological parameter with mass dimension 2. From the GMG action, one derives the Einstein equation

$$\begin{aligned} \sigma G_{\mu \nu }+\lambda _G g_{\mu \nu }+\frac{1}{2m^2}K_{\mu \nu }+\frac{1}{\mu }C_{\mu \nu }=0, \end{aligned}$$
(3.9)

where \(C_{\mu \nu }\) is given by Eq. (3.3) and \(K_{\mu \nu }\) takes the form

$$\begin{aligned} K_{\mu \nu }&= 2\nabla ^2R_{\mu \nu }-\frac{1}{2}\nabla _\mu \nabla _\nu R-\frac{1}{2}\nabla ^2Rg_{\mu \nu }\nonumber \\&+\,4R_{\mu \rho \nu \sigma }R^{\rho \sigma } -\frac{3}{2} R R_{\mu \nu }-R_{\rho \sigma }R^{\rho \sigma }g_{\mu \nu } +\frac{3}{8}{R}^2 g_{\mu \nu }. \end{aligned}$$
(3.10)

It is pointed out that the BTZ black hole solution (2.1) is allowed only for \(\lambda _G=\Lambda ^2/4m^2+\sigma \Lambda \). Using (2.3) and the TT gauge condition (2.8), the linearized equation of (3.9) can be written as

$$\begin{aligned} \Big [\bar{\nabla }^2-2\Lambda \Big ] \left[ \bar{\nabla }^2h_{\mu \nu } +\frac{m^2}{\mu }\epsilon _{\mu }^{~\alpha \beta }\bar{\nabla }_{\alpha } h_{\beta \nu }+\left( \sigma m^2-\frac{5}{2}\Lambda \right) h_{\mu \nu }\right] =0. \end{aligned}$$
(3.11)

Considering the above equation, we read off the second-order equation of the massive graviton

$$\begin{aligned} \bar{\nabla }^2h_{\mu \nu } +\frac{m^2}{\mu }\epsilon _{\mu }^{~\alpha \beta }\bar{\nabla }_{\alpha } h_{\beta \nu }+\left( \sigma m^2-\frac{5}{2}\Lambda \right) h_{\mu \nu }=0 \end{aligned}$$
(3.12)

which is further factorized into

$$\begin{aligned} \left[ \delta _{\mu }^{\beta }+ \frac{1}{m_+}\epsilon _{\mu }^{~\rho \beta }\bar{\nabla }_{\rho } \right] \left[ \delta _{\beta }^{\gamma } +\frac{1}{m_-}\epsilon _{\beta }^{~\sigma \gamma }\bar{\nabla }_{\sigma } \right] h_{\gamma \nu }=0. \end{aligned}$$
(3.13)

Here \(m_{\pm }\) take the forms

$$\begin{aligned} m_{\pm }=\frac{m^2}{2\mu }\pm \sqrt{\frac{m^4}{4\mu ^2}-\sigma m^2-\frac{\Lambda }{2}}. \end{aligned}$$
(3.14)

This implies that two massive gravitons with mass \(m_\pm \) are described by two first-order equations, respectively,

$$\begin{aligned} \epsilon _{\mu }^{~\alpha \beta }\bar{\nabla }_{\alpha }h_{\beta \nu } +m_+ h_{\mu \nu } =0,~~~\epsilon _{\mu }^{~\alpha \beta }\bar{\nabla }_{\alpha }h_{\beta \nu } +m_- h_{\mu \nu } =0. \end{aligned}$$
(3.15)

As squaring their first-order equations, acting two operations [\(\epsilon _{\sigma }^{~\rho \mu } \bar{\nabla }_{\rho }-m_+\delta _{\sigma }^{\mu }\)] and [\(\epsilon _{\sigma }^{~\rho \mu } \bar{\nabla }_{\rho }-m_-\delta _{\sigma }^{\mu }\)] on (3.15) leads to two second-order equations

$$\begin{aligned} \Big [\bar{\nabla }^2-2\Lambda -M_\mathrm{GMG+}^2\Big ]h_{\sigma \nu }=0,~~~\Big [\bar{\nabla }^2-2\Lambda -M_\mathrm{GMG-}^2\Big ]h_{\sigma \nu }=0, \end{aligned}$$
(3.16)

where

$$\begin{aligned} M_{\mathrm{GMG}\pm }^2=m_{\pm }^2-\frac{1}{\ell ^2}. \end{aligned}$$
(3.17)

Using the bound given by the stable condition (2.31), we have the mass bound

$$\begin{aligned} M_\mathrm{GMG \pm }^2 \ge -\frac{1}{\ell ^2} \rightarrow m_{\pm }^2 \ge 0. \end{aligned}$$
(3.18)

Consequently, the \(s\)-mode stability condition after squaring their first-order equations is given by

$$\begin{aligned} m^2_i \ge 0, \end{aligned}$$
(3.19)

where

$$\begin{aligned} m^2_1=\mu ^2(\mathrm{TMG}),~~~~m^2_2=m^2_+(\mathrm{GMG}),~~~~m^2_3=m^2_-(\mathrm{GMG}). \end{aligned}$$
(3.20)

3.3 Quasinormal mode analysis

We note that (3.5) and (3.15) belong to the first-order equation and they are parity-odd, while (3.6) and (3.16) are the second-order equation and are parity-even. Furthermore, (3.6) and (3.16) have ambiguities on the sign of mass. Hence, it would be better to use (3.5) and (3.15) than (3.6) and (3.16) when considering an another stability analysis for the parity-odd theories.

In this section, we redo the stability analysis of the massive graviton for TMG and GMG by computing quasinormal frequencies. For this purpose, we note that the type I and II perturbations are suitable for the second-order differential Eqs. (3.6) and (3.16), while these are inappropriate for applying to the first-order Eqs. (3.5) and (3.15) directly. As will be shown in Appendix, applying type I and II to (3.5) and (3.15) leads to all null perturbations due to the parity-oddness of their equations.

Therefore, we perform the stability analysis with solving (3.5) and (3.15) to find quasinormal frequencies by following the approach developed in [15]. We first note that (3.5) and (3.15) can be written as a single first-order equation

$$\begin{aligned} \epsilon _{\mu }^{~\alpha \beta }\bar{\nabla }_{\alpha }h_{\beta \nu } +m_i h_{\mu \nu } =0, \end{aligned}$$
(3.21)

where \(m_i\) denote \(\mu \) and \(m_{\pm }\). For our purpose, we consider a BTZ black hole metric with \(\mathcal{M}=1\) and \(\ell =1 (r_+=1)\) given in global coordinates

$$\begin{aligned} ds^2_\mathrm{gc}=\bar{g}_{\mu \nu }dx^{\mu }dx^{\nu }=-\sinh ^2\rho dt^2+\cosh ^2\rho d\phi ^2+d\rho ^2, \end{aligned}$$
(3.22)

which is obtained by replacing \(r\) by \(r=\cosh \rho \) in (2.1). In what follows we use light-cone coordinates with \(u=t+\phi \) and \(v=t-\phi \) as

$$\begin{aligned} ds^2_\mathrm{lc}=\bar{g}_{\mu \nu }dx^{\mu }dx^{\nu }=\frac{1}{4}du^2+\frac{1}{4}dv^2 -\frac{1}{2}\cosh 2\rho dudv+d\rho ^2. \end{aligned}$$
(3.23)

The metric (3.23) admits isometry group of SL(2,R)\(\times \)SL(2,R), which allows us two sets of the Killing vector fields, \(L_{0,\pm 1}\) and \(\bar{L}_{0,\pm 1}\) given as

$$\begin{aligned} \begin{aligned} L_0&=-\partial _u,\\ L_{-1}&=e^{-u}\left( -\frac{\cosh 2\rho }{\sinh 2\rho }\partial _u -\frac{1}{\sinh 2\rho }\partial _v-\frac{1}{2}\partial _{\rho }\right) ,\\ L_{1}&=e^{u}\left( -\frac{\cosh 2\rho }{\sinh 2\rho }\partial _u -\frac{1}{\sinh 2\rho }\partial _v+\frac{1}{2}\partial _{\rho }\right) \end{aligned} \end{aligned}$$
(3.24)

and \(\bar{L}_{0,\pm 1}\) are defined by operation of \(u\leftrightarrow v\) in (3.24). Three vector fields \(L_{0,\pm 1}\) satisfy the SL(2,R) algebra

$$\begin{aligned}{}[L_0,L_{\pm 1}]=\mp L_{\pm 1},~~~[L_1,L_{-1}]=2L_0. \end{aligned}$$
(3.25)

Taking the perturbation ansatz in terms of \((u,v,\rho )\)

$$\begin{aligned} h_{\mu \nu }=e^{-i\omega t-ik\phi }\psi _{\mu \nu }(\rho ) =e^{-ip_+ u-ip_- v}\psi _{\mu \nu }(\rho ),~~~p_{\pm }=\frac{1}{2}(\omega \pm k), \end{aligned}$$
(3.26)

the \(s\)-mode (\(k=0\)) solutions to the first-order Eq. (3.21) are given by right(\(r\))/left(\(l\)) moving modes:

$$\begin{aligned} h^{r}_{\mu \nu }\!=\!e^{-2h_r t}\psi _{\mu \nu }\!=\! e^{-2h_r t}(\sinh \rho )^{-2h_r}\left( \begin{array}{c@{\quad }c@{\quad }c} 1 &{} 0 &{} \frac{2}{\sinh 2\rho } \\ 0 &{} 0 &{} 0 \\ \frac{2}{\sinh 2\rho } &{} 0 &{} \frac{4}{\sinh ^22\rho } \end{array} \right) \,,~~~h_r\!=\!-\frac{1}{2}(m_i\!+\!1) \nonumber \\ \end{aligned}$$
(3.27)

for \(p_-=-ih_r\) and

$$\begin{aligned} h^{l}_{\mu \nu }=e^{-2h_lt}\psi _{\mu \nu }= e^{-2h_lt}(\sinh \rho )^{-2h_l}\left( \begin{array}{c@{\quad }c@{\quad }c} 0 &{} 0 &{} 0 \\ 0 &{} 1 &{} \frac{2}{\sinh 2\rho } \\ 0 &{} \frac{2}{\sinh 2\rho } &{} \frac{4}{\sinh ^22\rho } \end{array} \right) \,,~~~h_l=\frac{1}{2}(m_i-1) \nonumber \\ \end{aligned}$$
(3.28)

for \(p_+=-ih_l\). Note that the solution (3.27) satisfies the chiral highest weight condition of \(\bar{L}_1h_{\mu \nu }=0\) and (3.28) satisfies the anti-chiral highest weight condition of \(L_1h_{\mu \nu }=0\) which are equivalent to the transversality condition of \(\bar{\nabla }_{\mu }h^{\mu }_{~\nu }=0\). However, requiring both conditions leads to null modes. Considering relations \(p_\pm =\omega ^{r/l}/2\), the corresponding quasinormal frequencies, whose quasinormal modes satisfy the boundary conditions: ingoing modes at the horizon and Dirichlet boundary condition at infinity, can be written as

$$\begin{aligned} \omega ^r&= -2ih_r=2i \Big (\frac{m_i}{2}+\frac{1}{2}\Big ),\end{aligned}$$
(3.29)
$$\begin{aligned} \omega ^l&= -2ih_l=2i \Big (-\frac{m_i}{2}+\frac{1}{2}\Big ). \end{aligned}$$
(3.30)

The complete tower of right- and left-moving quasinormal modes is generated by acting \(L_{-1}\bar{L}_{-1}\) on \(h^{r/l}_{\mu \nu } n\) times as

$$\begin{aligned} h^{(n),~ r/l}_{\mu \nu } =\Big (L_{-1}\bar{L}_{-1}\big )^nh^{r/l}_{\mu \nu }, \end{aligned}$$
(3.31)

which leads to their quasinormal frequencies with overtone number \(n\)

$$\begin{aligned} \omega _{n}^{r}&= -2i(h_r+n),\end{aligned}$$
(3.32)
$$\begin{aligned} \omega _{n}^{l}&= -2i(h_l+n). \end{aligned}$$
(3.33)

Since the stability condition is determined by two basic quasinormal frequencies with \(\omega ^{r/l}_\mathrm{I}<0\) for \(\omega ^{r/l}=\omega _{\mathrm{R}}^{r/l}+i\omega _\mathrm{I}^{r/l}\), it is given by

$$\begin{aligned} |m_i|>1/\ell , \end{aligned}$$
(3.34)

where the AdS\(_3\) curvature radius \(\ell \) is restored for convenience. It seems, however, that there is some discrepancy between (3.34) and (3.19).

4 Discussions

In this work, we have established the stability of the massive graviton around BTZ black hole in massive gravity theories which are classified into the parity-even gravity theories (NMG, SDG) and the parity-odd theories (TMG, GMG). For the parity-even massive gravities, the stability conditions employed by the \(s\)-mode analysis are exactly the same with the BF-bound, which corresponds to \(M_i^2\ge -1/\ell ^2\) (2.29). For the parity-even gravity theories, the \(s\)-mode analysis and the BF-bound based on the second-order massive equation are consistent with the Birmingham–Mokhtari–Sachs result requiring the boundedness of perturbation at the horizon where a second-order hypergeometric equation was used. These are given by the condition of \(m^2_i \ge 0 ~(|m_i|\ge 0)\).

We stress that the stability analysis performed by the quasinormal frequencies gave a condition of \(|m_i|>1/\ell \), being different from \(|m_i|\ge 0\). We may interpret it by mentioning that the connection between potential and quasinormal frequencies condition is guaranteed if the second-order equation is used as Schrödinger equation [5, 6, 27]. We here have obtained the quasinormal frequencies by using the first-order equation. Hence, the condition of \(|m_i|>1/\ell \) based on quasinormal modes does not comprise the stability condition \(|m_i| \ge 0\) obtained by solving the second-order equation. In this case, the unstable quasinormal modes exist for \(0\le |m_i|\le 1/\ell \). It is, however, pointed out that these unstable modes may be truncated by requiring the boundedness at the horizon when considering the generalized boundary conditions at asymptotic infinity [12]. Hence, it suggests that the stability condition might be extended to comprise

$$\begin{aligned} |m_i|\ge 0. \end{aligned}$$
(4.1)

Finally, we would like to mention that the stability of a black hole in four-dimensional massive gravity is determined by the Gregory-Laflamme instability of a five-dimensional black string. It turned out that the small Schwarzschild black hole in the dRGT massive gravity [28] and fourth-order gravity [29] is unstable against the metric and Ricci tensor perturbations. In the present work, the stability was mainly determined by the asymptotes of black hole spacetimes. Hence, it suggests for a future direction that the stability of the massive graviton around a BTZ black hole would be revisited by using the (in)stability of four-dimensional black string.