1 Introduction

Since the time of Poincaré, invariant objects are very much important to understand the behavior of a dynamical system, especially, phase space. Moreover, there are many possible approaches to find the invariant objects, whereas the normal forms (truncated) are very useful because these can give integrable approximations to the dynamics under appropriate hypothesis [15]. Because of the approximation of true dynamics by the normal forms, invariant objects of the initial system get approximated also, accordingly [17, 36]. The approximate first integrals are those quantities, which are almost preserved through the system’s flow. This shows that the surface levels by the flow are almost invariant. Some informations about the dynamics can be obtained through this property. To minimize the overflow and complexity in the computations, an appropriate approach is to use of power series or Fourier sires, or a combination of both to represent the object. Because in many cases they needed only a few numbers of terms to maintain the good accuracy. Some other approach can also be found in Gómez et al. [10], Jorba and Masdemont [16], in which trigonometric series is used. The normal forms of the Hamiltonian system up to some finite order is necessary to study the nonlinear stability of the equilibrium points using Arnold–Moser theorem in non-resonance case. They also help to know the behavior of dynamics in the neighborhood of the invariant objects. Many researchers have described the different method to find the normal forms of the Hamiltonian of the dynamical system [2, 6, 8, 15, 19, 29, 39]. In the normal forms, the central idea is to find suitable transforms of the phase co-ordinates, which can convert the Hamiltonian system in its simplest form up to a finite order of accuracy. Normalization of Hamiltonian is obtained to change the Hamiltonian into its simplest form using the method of Lie transforms [6, 15].

Because of radiating primary in the present problem under the analysis, force due to radiation pressure came into existence [32, 35], which acts in opposite direction to the gravitational attraction force of the primary. Concept of Poynting–Roberston drag is came into the picture when, Poynting [30] investigated the effect of radiation pressure on the moving particle in interplanetary space and Robertson [34] modified the Poynting’s theory through the principle of relativity. In the analysis of Roberston, he considered only first order terms in the expression related to the ratio of velocity of the particle to that of the light. The radiation force is expressed as

$$\begin{aligned} \vec {F}=F_{p}\left( \frac{\vec {R}}{R}-\frac{\vec {V}.\vec {R}\vec {R}}{cR^{2}}-\frac{\vec {V}}{c}\right) , \end{aligned}$$
(1)

where \(F_p\) is the radiation pressure force due to radiating primary; \(\vec {R}\) is the position vector of the particle relative to the radiating primary; \(\vec {V}\) is the velocity of the particle; and c is the speed of the light. First term of the Eq. (1) denotes the radiation pressure, second term represents the Doppler shift due to the motion of the particle, whereas third term corresponds to the absorption and subsequent re-emission part of induced radiation. The combined form of the last two terms of the Eq. (1) known as Poynting–Robertson (P–R) drag. Chernikov [4] analyzed the photogravitational restricted three body problem (RTBP) with P–R drag under the frame of Sun-planet-particle system and found that non-collinear (triangular) equilibrium points are unstable. Effect of P–R drag including radiation pressure is described by Schuerman [35]. A similar analysis is presented by Murray [28] and Ragos and Zafiropoulos [31] to observed the effect of P–R drag in the context of existence and stability of the equilibrium points. Kushvah et al. [20] examined the nonlinear stability in the generalized photogravitational RTBP with P–R drag of first primary and oblateness of secondary and found that triangular equilibrium points are unstable, whereas Mishra and Ishwar [27] investigated about the stability of non-collinear equilibrium points in the photogravitational elliptic RTBP with P–R drag. Kushvah et al. [21] and Kishor and Kushvah [18] have analyzed the effect of radiation pressure force on the existence and linear stability of the equilibrium points in the generalized photogravitational Chermnykh-like problem with a disc. They found that the effect of perturbation factors are significant. In literature, many researchers have analyzed the photogravitational RTBP in nonlinear sense by considering one or two perturbations at a time [1, 13, 22, 24, 38], but very few of them have considered the problem under the combined influence of few perturbations [19, 20]. Ishwar and Sharma [14] have discussed about the nonlinear stability of out of plane equilibrium points in the RTBP with oblate primary and found that \(L_6\) point is stable in nonlinear sense. Raj and Ishwar [33] have obtained diagonalized form of the Hamiltonian with P–R drag. Kishor and Kushvah [19] have studied nonlinear stability of triangular equilibrium points in the Chermnykh-like problem, in the presence of radiation pressure, oblateness and a disc. They found that these perturbations affect the numerical results significantly.

Due to above reasons in addition to wide applications of the RTBP in mission design, we are motivated to study the problem under the influence of the radiation pressures and P–R drags of both primary and secondary. In the present study, we are interested to compute the fourth order normalized Hamiltonian and utilizing them to analyze the nonlinear stability of triangular equilibrium points using Arnold–Moser theorem in non-resonance case. Because of both primary and secondary radiating, the problem under analysis includes the four perturbing parameters in the form of mass reduction factors \(q_1,\,q_2\) due to the radiation pressures of the primaries and P–R drags \(W_1,\,W_2\) of both the primaries, respectively. The paper is organized as follows: In Sect. 2, we have formulated the problem and found the equations of motion. Section 3 presents the second order normalized Hamiltonian of the problem under analysis. Nonlinear stability analysis is discussed in Sect. 4. Section 5 is devoted to Birkhoff normal form and application of Arnold–Moser theorem in non-resonance case. Results are concluded in Sect. 6. For algebraic and numerical computations, Mathematica® [40] software package is used. The results of this study may be used to describe more generalized problem under the influence of other perturbations such as albedo, solar wind drag, Stokes drag etc. [12, 37].

2 Mathematical Formulation

We consider the photogravitational restricted three body problem with P–R drag, which consists of motion of an infinitesimal mass under the influence of gravitational field and radiation effect of two massive and radiating bodies of masses \(m_1\) and \(m_2,\, (m_1>m_2)\), respectively, called primaries. Forces, which govern the motion of infinitesimal mass are gravitational attractions, radiation pressures and P–R drags of both the primaries, respectively. It is assumed that gravitational effect of infinitesimal mass on the system is negligible. Units are normalized such as units of mass and distance are taken as the sum of the masses of both the primaries and separation distance between them, respectively, whereas unit of time is the time period of the rotating frame. We suppose that the coordinate of the primaries are \((-\mu ,\,0),\, (1-\mu ,\, 0),\) respectively and that of infinitesimal mass is \((x,\,y)\), then the equations of motion [33] are

$$\begin{aligned} \ddot{x}-2\dot{y}= & {} \frac{\partial U}{\partial x}, \end{aligned}$$
(2)
$$\begin{aligned} \ddot{y}-2\dot{x}= & {} \frac{\partial U}{\partial y}, \end{aligned}$$
(3)

where

$$\begin{aligned} \frac{\partial U}{\partial x}= & {} x-\frac{q_1(1-\mu )(x+\mu )}{r_{1}^{3}}-\frac{q_2(1-\mu )(x+\mu -1)}{r_{2}^{3}}\nonumber \\&-\frac{W_1S_1}{r_{1}^2}-\frac{W_2S_2}{r_{2}^2}, \end{aligned}$$
(4)
$$\begin{aligned} \frac{\partial U}{\partial y}= & {} y-\frac{q_1(1-\mu )y}{r_{1}^{3}}-\frac{q_2(1-\mu )y}{r_{2}^{3}}-\frac{W_1S_3}{r_{1}^2}-\frac{W_2S_4}{r_{2}^2}, \end{aligned}$$
(5)

and further

$$\begin{aligned} S_1= & {} \frac{(x+\mu )\{(x+\mu )\dot{x}+y\dot{y}\}}{r_{1}^2}+\dot{x}-y,\\ S_2= & {} \frac{(x+\mu -1)\{(x+\mu -1)\dot{x}+y\dot{y}\}}{r_{2}^2}+\dot{x}-y,\\ S_3= & {} \frac{y\{(x+\mu )\dot{x}+y\dot{y}\}}{r_{1}^2}+\dot{y}+x+\mu ,\\ S_4= & {} \frac{y\{(x+\mu -1)\dot{x}+y\dot{y}\}}{r_{2}^2}+\dot{y}+x+\mu -1 \end{aligned}$$

with \(q_i=1-F_{pi}/F_{gi},\,i=1,2\) as mass reduction factors of both the primaries, respectively; \(F_{pi},\,F_{gi},\, i=1, 2\) are the forces of radiation pressure and gravitational attraction of the respective primaries; \(W_1=[(1-q_1)(1-\mu )]/c_d\) and \(W_2=[(1-q_2)\mu ]/{c_d}\) as P–R drags of both the primaries, respectively; \(c_{d}\) is the speed of light in non-dimensional form; \(r_{1},\, r_{2}\) are distances of infinitesimal mass from the first and second primary, which are given as

$$\begin{aligned} r_{1}^2=(x+\mu )^2+y^2,\quad r_{2}^2=(x+\mu -1)^2+y^2. \end{aligned}$$
(6)

The co-ordinates \((x_0,\,\pm y_0)\) of triangular equilibrium points \(L_{4,5}\) are obtained on similar basis as in Raj and Ishwar [33]. To overcome the complexity in the analysis, co-ordinates \(x_0\) and \(y_0\) are linearized with respected to \(W_1,\,W_2,\epsilon _1,\,\epsilon _2\), keeping in mind that the perturbing parameters lie in \((0,\,1)\) so, take \(q_1=1-\epsilon _1,\,q_2=1-\epsilon _2\), where \(\epsilon _1,\,\epsilon _2\) are very small. The linearized co-ordinates \(x_0\) and \(y_0\) are

$$\begin{aligned} x_0= & {} \frac{1}{2}-\mu -\frac{4W_1(2-\mu )}{3\sqrt{3}}- \frac{4W_2(1+2\mu )}{3\sqrt{3}}-\frac{\epsilon _1}{3}+\frac{\epsilon _2}{3}, \end{aligned}$$
(7)
$$\begin{aligned} y_0= & {} \pm \left[ \frac{\sqrt{3}}{2}+\frac{4W_1(2-3\mu )}{9}+\frac{4W_2(1-3\mu )}{9}-\frac{\epsilon _1}{3\sqrt{3}}- \frac{\epsilon _2}{3\sqrt{3}}\right] . \end{aligned}$$
(8)

The plus sign corresponds to \(L_4\), whereas minus sign corresponds to \(L_5\). The Hamiltonian function of the problem is written as

$$\begin{aligned} H= & {} p_x\dot{x}+p_y\dot{y}-\frac{\dot{x}^2+\dot{y}^2}{2}-\frac{x^2+y^2}{2}- x\dot{y}\nonumber \\&+\dot{x}y-\frac{(1-\mu )q_1}{r_1}-\frac{\mu q_2}{r_2}-W_1S_5-W_2S_6, \end{aligned}$$
(9)

where

$$\begin{aligned} S_5= & {} \frac{(x+\mu )\dot{x}+y\dot{y}}{2r_{1}^2}-\arctan {\left( \frac{y}{x+\mu }\right) },\\ S_6= & {} \frac{(x+\mu -1)\dot{x}+y\dot{y}}{2r_{1}^2}-\arctan {\left( \frac{y}{x+\mu -1}\right) }. \end{aligned}$$

The conjugate momenta \(p_x,\,p_y\) corresponding to generalized co-ordinate \(x,\,y\) respectively, are given as

$$\begin{aligned} p_x= & {} \dot{x}-y+\frac{W_1(x+\mu )}{2r_{1}^2}+\frac{W_2(x+\mu -1)}{2r_{2}^2}, \end{aligned}$$
(10)
$$\begin{aligned} p_y= & {} \dot{y}+x+\frac{W_1 y}{2r_{1}^2}+\frac{W_2 y}{2r_{2}^2}. \end{aligned}$$
(11)

3 Second Order Normal Form of the Hamiltonian

In the present analysis only the stability of \(L_4\) is analyzed, because the dynamics of \(L_5\) is similar to that of \(L_4\). Only first order terms in the perturbing parameters \(W_1,\,W_2,\,q_1,\,q_2\) are considered for simplifying the complex calculations involved in the problem through out the analysis. The second order normal form of the Hamiltonian of the problem under analysis is obtained in Raj and Ishwar [33] and for self sufficiency of this paper, we have taken some necessary expressions in appropriate form there to use under this section. Shifting the origin to the triangular equilibrium point \(L_4\) using simple transformations as

$$\begin{aligned}&x^*=x-x_0,\, y^*=y-y_0,\,\\&{p_x}^*=p_x+y_0,\, {p_y}^*=p_y-x_0. \end{aligned}$$
Fig. 1
figure 1

Variation of critical mass ratio \(\mu _c\) with respect to a\(W_1\), b\(W_2\), c\(\epsilon _1\) and d\(\epsilon _2\)

Full size image

Substituting these variables in Hamiltonian (9), we get new Hamiltonian \(H^*\). Now, expanding the new Hamiltonian using Taylor’s series about the origin, which is now, the triangular equilibrium point, \(H^*\) can be written as

$$\begin{aligned} H^*=H_{0}^*+H_{1}^*+H_{2}^*+H_{3}^*+\cdots +H_{n}^*+\cdots , \end{aligned}$$
(12)

where

$$\begin{aligned} H_{n}^*=\sum H_{ijkl}{x^*}^i{y^*}^j{{p_x}^*}^k{{p_y}^*}^l, \end{aligned}$$
(13)

such that \(i+j+k+l=n\). Since, the origin is the triangular equilibrium point, \(H_{1}^*\) must vanish, whereas \(H_{0}^*\) is constant, and hence, it can be dropped out as it is irrelevant to the dynamics. The quadratic Hamiltonian \(H_{2}^*\), which is to be normalized first and then to be used for higher order normalization, is given as

$$\begin{aligned} H_{2}^*=\frac{{{p_x}*}^2+{{p_y}^*}^2}{2}+y^*{p_x}^*-x^*{p_y}^*+E{x^*}^2+Gx^*y^*+F{y^*}^2, \end{aligned}$$
(14)

where

$$\begin{aligned} E= & {} \frac{1}{8}+\frac{4W_1}{\sqrt{3}}+\frac{2W_1}{\sqrt{3}}+\frac{\epsilon _1}{4} -\frac{\epsilon _2}{2}, \end{aligned}$$
(15)
$$\begin{aligned} F= & {} -\frac{5}{8}-\frac{4W_1}{\sqrt{3}}-\frac{2W_1}{\sqrt{3}}-\frac{\epsilon _1}{4} +\frac{\epsilon _2}{2}, \end{aligned}$$
(16)
$$\begin{aligned} G= & {} -\gamma \left( 1-\frac{32W_1}{9\sqrt{3}}-\frac{16W_1}{9\sqrt{3}}-\frac{2\epsilon _1}{9}+\frac{4\epsilon _2}{9}\right) , \end{aligned}$$
(17)
$$\begin{aligned} \text {with}\quad \gamma= & {} \frac{3\sqrt{3}}{4}(1-2\mu ). \end{aligned}$$
(18)

In the present study, the problem is dealt with four perturbation parameters in the form of P–R drag and radiation pressure of both the primaries. Hence, the coefficient \(H_{ijkl}\) for \(i,\,j,\,k,\,l=0,\,1,\,2,\,3,\,4\) such that \(i+j+k+l=4\) in (13) can be bifurcated into five parts such as \(H_{ijkl1},\,H_{ijkl2},\,H_{ijkl3},\,H_{ijkl4}\), and \(H_{ijkl5}\), which corresponds to the terms in classical case, terms with P–R drag of first primary \(W_1\), P–R drag of second primary \(W_2\), radiation pressure of first primary \(\epsilon _1=1-q_1\) and radiation pressure of second primary \(\epsilon _2=1-q_2\), respectively. Thus,

$$\begin{aligned} H_{ijkl}=H_{ijkl1}+H_{ijkl2}+H_{ijkl3}+H_{ijkl4}+H_{ijkl5}. \end{aligned}$$
(19)

It is noted that if there is no perturbations in the system, i.e. \(W_1=W_2=\epsilon _1=\epsilon _2=0\), then \(H_{ijkl}=H_{ijkl1}\), which is nothing but the coefficient of the Hamiltonian in classical case.

Hamiltonian equations of motion of the infinitesimal mass in matrix form is written as

$$\begin{aligned} \begin{bmatrix}\dot{x^*}\\ \dot{y^*}\\ \dot{{p_x}^*}\\ \dot{{p_y}^*}\end{bmatrix}= \begin{bmatrix}0&\qquad 1&\qquad 1&\qquad 0\\ -1&\qquad 0&\qquad 0&\qquad 1\\ -2E&\qquad -G&\qquad 0&\qquad 1\\ -G&\qquad -2F&\qquad -1&\qquad 0\end{bmatrix} \begin{bmatrix}x^*\\ y^*\\ {p_x}^*\\ {p_y}^*\end{bmatrix}. \end{aligned}$$
(20)

The characteristic equation of the system (20) is

$$\begin{aligned} \lambda ^4+2(E+F+1)\lambda ^2+(4EF-G^2-2E-2F+1)=0. \end{aligned}$$
(21)

Solving the simplified discriminant of the characteristic equation (21) as

$$\begin{aligned} (E+F+1)^2-(4EF-G^2-2E-2F+1)=0, \end{aligned}$$
(22)

we have the value of critical mass ratio \(0<\mu _c\le (1/2)\) as

$$\begin{aligned} \mu _c=0.0385209+0.0823761 W_1+0.0823761 W_2+0.0178349 \epsilon _1-0.356699 \epsilon _2,\nonumber \\ \end{aligned}$$
(23)

which is similar to that of Kushvah et al. [20] and Kishor and Kushvah [18] and agree with the classical value \(\mu _c=0.0385209\). Figure 1a–d shows the variations of critical mass ratio \(\mu _c\) with respect to perturbing parameters \(W_1,\,W_2,\, \epsilon _1\) and \(\epsilon _2,\) respectively. We observed that the effects of the perturbations in question are significant. As, system will be stable when four roots of the characteristic equation (21) are pure imaginary, which is possible when the mass parameter \(\mu \) satisfy the condition \(0<\mu <\mu _c\). Since, we are analyzing the nonlinear stability within the range of linear stability \(0<\mu <\mu _c\), it is obvious to assume that roots of the characteristic equation (21) are pure imaginary. Suppose, the roots of the characteristic equation (21) are \(\pm i\omega _1\) and \(\pm i\omega _2\), where \(\omega _1,\,\omega _2\) can be obtained by solving the equation

$$\begin{aligned} \omega ^4-2(E+F+1)\omega ^2+(4EF-G^2-2E-2F+1)=0. \end{aligned}$$
(24)

Motion corresponds to frequencies \(\omega _{1},\,\omega _2\in \mathbb {R}\) are known as long and short periodic motion of infinitesimal mass at \(L_4\) with periods of \(2\pi /\omega _1\) and \(2\pi /\omega _2\), respectively. Frequencies \(\omega _{1},\,\omega _2\) corresponding to the long and short periodic motion are related to each other by the relations

$$\begin{aligned} \omega _{1}^2+\omega _{2}^2= & {} 2E+2F+2, \end{aligned}$$
(25)
$$\begin{aligned} \omega _{1}^2\omega _{2}^2= & {} 4EF-G^2-2E-2F+1. \end{aligned}$$
(26)

Substituting the values of \(E,\,F\) and G from Eqs. (15–17), we get

$$\begin{aligned} \omega _{1}^2+\omega _{2}^2= & {} 1, \end{aligned}$$
(27)
$$\begin{aligned} \omega _{1}^2\omega _{2}^2= & {} \frac{27}{16}\gamma ^2-4\sqrt{3}W_1-2\sqrt{3}W_2-\frac{2\epsilon _1}{3}+\frac{3\epsilon _2}{4}, \end{aligned}$$
(28)

where the values of \(\omega _1\) and \(\omega _2\) are

$$\begin{aligned} \omega _1=\sqrt{-1+\sqrt{1-4\delta }},\quad \omega _2=\sqrt{-1-\sqrt{1-4\delta }}, \end{aligned}$$
(29)

with

$$\begin{aligned} \delta =\frac{27}{16}-\gamma ^2-4\sqrt{3}W_1-2\sqrt{3}W_2-\frac{3\epsilon _1}{4}+ \frac{3\epsilon _2}{2}. \end{aligned}$$
(30)

The real normalized Hamiltonian of the Hamiltonian (14) up to second order is given as [33]

$$\begin{aligned} H_2=\omega _1\frac{{\mathbf {x^2+p_{x}^2}}}{2}+\omega _2\frac{{\mathbf {y^2+p_{y}^2}}}{2}, \end{aligned}$$
(31)

which is complexified by using the co-ordinate transformations

$$\begin{aligned} \mathbf {x}= & {} \frac{X+iP_X}{\sqrt{2}}, \end{aligned}$$
(32)
$$\begin{aligned} \mathbf {y}= & {} \frac{-Y+iP_Y}{\sqrt{2}}, \end{aligned}$$
(33)
$$\begin{aligned} \mathbf {p_x}= & {} \frac{iX+P_X}{\sqrt{2}}, \end{aligned}$$
(34)
$$\begin{aligned} \mathbf {p_y}= & {} \frac{iY-P_Y}{\sqrt{2}} \end{aligned}$$
(35)

and changed as

$$\begin{aligned} H_2=i\omega _1 XP_X-i\omega _2YP_Y, \end{aligned}$$
(36)

Finally, symplectic matrix \(\mathbf {C}\) of the symplectic transformations, which are used to obtain the complex normal form of Hamiltonian is given as [33]

$$\begin{aligned}&\mathbf{{C}}=\begin{bmatrix}s_{ij}\end{bmatrix},\,1\le i,j\le 4 \end{aligned}$$
(37)

with

$$\begin{aligned}&s_{11}=0=s_{12},\,s_{13}=\frac{1-2F+\omega _{1}^2}{\sqrt{d(\omega _1)}},\\&s_{14}=\frac{1-2F+\omega _{1}^2}{\sqrt{d(\omega _2)}},\,s_{21}=\frac{2\omega _1}{\sqrt{d(\omega _1)}},\\&s_{22}=\frac{2\omega _1}{\sqrt{d(\omega _2)}},\, s_{23}=\frac{G}{\sqrt{d(\omega _1)}},\\&s_{24}=\frac{G}{\sqrt{d(\omega _2)}},\, s_{31}=\frac{\omega _{1}^3-(2F+1)\omega _1}{\sqrt{d(\omega _1)}},\\&s_{32}=\frac{\omega _{2}^3-(2F+1)\omega _2}{\sqrt{d(\omega _2)}},\, s_{33}=\frac{-G}{\sqrt{d(\omega _1)}},\\&s_{34}= \frac{-G}{\sqrt{d(\omega _2)}},\, s_{41}=\frac{G\omega _1}{\sqrt{d(\omega _1)}},\, s_{42}=\frac{G\omega _2}{\sqrt{d(\omega _2)}},\\&s_{43}=\frac{1-2F-\omega _{1}^2}{\sqrt{d(\omega _1)}},\, s_{44}=\frac{1-2F-\omega _{2}^2}{\sqrt{d(\omega _2)}}, \end{aligned}$$

where \(d(\omega _i)\) for \(i=1,\,2\) is obtained from the following equation

$$\begin{aligned} d(\omega )=\omega \left[ \omega ^4-(2E+6F)\omega ^2+(4EF+4F^2-2E+2F-2)\right] . \end{aligned}$$
(38)

4 Nonlinear Stability in Non-resonance Case

Nonlinear stability of the equilibrium points can be described in two cases, one as resonance case and other as non-resonance case. For resonance case, the nonlinear stability is studied through the theorems of Markeev and Sokolskii [23] as in Goździewski [11] and for non-resonance case, it is analyzed through the Arnold–Moser theorem. In the present analysis the nonlinear stability of the perturbed triangular equilibrium point in non-resonance case will be studied through Arnold–Moser theorem [25, 26], which is described as follows:

Consider the Hamiltonian expressed in action variables \( I_1,\, I_2\) and angles variables \(\phi _1,\, \phi _2\) as,

$$\begin{aligned}&K=K_2+ K_4 + \cdots +K_{2m}+K_{2m+1}, \end{aligned}$$
(39)

in which: (i) \(K_{2m}\) is homogeneous polynomial of degree m in action variables \(I_1,\,I_2\) and \(K_{2m+1}\) is higher degree polynomial than m (ii) \(K_2=\omega _1 I_1-\omega _2 I_2\) with \(\omega _{1,2}\) as positive constants (iii) \(K_4=-(AI_{1}^2+BI_1I_2+CI_{2}^2)\), where \(A,\,B,\,C\) are constants to be determined. Since, \(K_2,\,K_4,\,\dots ,K_{2m}\) are functions of \(I_1\) and \(I_2\), the Hamiltonian (39) follows the Birkhoff normal form [2] up to the terms m. This can be obtained with some non-resonance condition on the frequencies \(\omega _{1},\,\omega _2\). To state the Arnold–Moser theorem, we assume that K is in the required form.

Arnold–Moser Theorem:The origin is stable for the system whose Hamiltonian is (39) provided for some\(\nu ,\,\,2\le \nu \le m\), \(D_{2\nu } = K_{2\nu }(\omega _2,\omega _1) \ne 0\).

Since, for Arnold–Moser theorem, Birkhoff normal form of the Hamiltonian is necessary and for Birkhoff normal form, assumption of non-resonance on frequencies is required. The non-resonance condition of frequencies as in Deprit and Deprit-Bartholome [9], Kishor and Kushvah [19] is that if \(\omega _{1},\,\omega _2\) are frequencies of infinitesimal mass in linear dynamics and \(\sigma \in \mathbb {Z}\) such that \(\sigma \ge 2\), then

$$\begin{aligned} \sigma _1\omega _1+\sigma _2\omega _2\ne 0 \end{aligned}$$
(40)

for all \(\sigma _{1},\,\sigma _2\in \mathbb {Z}\) satisfying \(|\sigma _1|+|\sigma _2|\le 2\sigma \). This is also, called as condition of irrationality, which insures that there exists a symplectic normalizing transformation which transform the Hamiltonian (12) in the form of Hamiltonian (39). Coefficients of the normalized Hamiltonian are independent on the integer \(\sigma \) as well as to the transformation obtained. In specific

$$\begin{aligned} \text {det}\begin{vmatrix}\frac{\partial ^2K}{\partial I_{1}^2}&\frac{\partial ^2K}{\partial I_1\partial I_{2}}&\frac{\partial K}{\partial I_{1}}\\\frac{\partial ^2K}{\partial I_2\partial I_{1}}&\frac{\partial ^2K}{\partial I_{2}^2}&\frac{\partial K}{\partial I_{2}}\\\frac{\partial K}{\partial I_{1}}&\frac{\partial K}{\partial I_{2}}&0\end{vmatrix}_{I_1,I_2=0} \end{aligned}$$
(41)

is invariant of the Hamiltonian (39) with respect to the symplectic transformation considered. The nonlinear stability of perturbed triangular equilibrium points is analyzed through the Arnold–Moser theorem under these conditions. In classical case frequencies \(\omega _{1},\,\omega _2\) satisfy the condition \(0<\omega _2<(1/\sqrt{2})<\omega _1<1\). Therefore, if \(\sigma =2\), then irrationality condition (40) fails for following pairs of integers \(\sigma _1=1,\, \sigma _2=-2\), \(\sigma _1=-1,\, \sigma _2=2\), \(\sigma _1=1,\, \sigma _2=-3\) and \(\sigma _1=-1,\, \sigma _2=3\). First, two pairs of integers with condition (40) yield \((\omega _1/\omega _2)=(1/2)\) and last two pairs of integers give \((\omega _1/\omega _2)=(1/3)\), which are also known as second and third order resonance of the frequencies respectively. If \((\omega _1/\omega _2)=(1/2)\) or \(\omega _1=2\omega _2\), then from Eqs. (27–28), we get

$$\begin{aligned} \frac{4}{25}=\frac{27}{16}\gamma ^2-4\sqrt{3}W_1-2\sqrt{3}W_2-\frac{2\epsilon _1}{3} +\frac{3\epsilon _2}{4}. \end{aligned}$$
(42)

Simplifying Eq. (42), we have a quadratic equation in \(\mu \) as

$$\begin{aligned} \frac{27}{16}\mu ^2-\frac{27}{16}\mu +\left( \sqrt{3}W_1+\frac{\sqrt{3}W_2}{2}+ \frac{3\epsilon _1}{16}-\frac{3\epsilon _2}{8}+\frac{1}{25}\right) =0. \end{aligned}$$
(43)

The solution \(\mu =\mu _{c1}\) of Eq. (43) within the stability range \(0<\mu <\mu _c\) is

$$\begin{aligned} \mu _{c1}=0.0242939+1.078820 W_1+0.539409 W_2+0.116785 \epsilon _1-0.233571 \epsilon _2.\nonumber \\ \end{aligned}$$
(44)

This means, Arnold–Moser theorem fails at \(\mu _{c1}\in (0,\,\mu _c)\). If \((\omega _1/\omega _2)=(1/3)\) or \(\omega _1=3\omega _2\), then proceeding on similar basis, we find that Arnold–Moser theorem fails at \(\mu =\mu _{c2}\), where

$$\begin{aligned} \mu _{c2}=0.013516+1.054920 W_1+0.527459 W_2+0.114198 \epsilon _1-0.228396 \epsilon _2.\nonumber \\ \end{aligned}$$
(45)

Equations (44–45) are similar to that of the results in Deprit and Deprit-Bartholome [9], Kishor and Kushvah [18] and agree with classical result in the absence of perturbing parameters. To see the effects of perturbing parameters on \(\mu _{c1}\) and \(\mu _{c2}\), its numerical values are computed and presented in Table 1. From Table 1, it is clear that the values of \(\mu _{c1}\) and \(\mu _{c2}\) are very much affected from radiation pressures and P–R drags of the primaries.

Table 1 \(\mu _{c1}\) and \(\mu _{c2}\) at different values of perturbing parameters
Full size table

5 Fourth Order Normalized Hamiltonian

Since, Birkhoff’s normal form up to fourth order of the Hamiltonian is necessary to apply the Arnold–Moser theorem, which is computed from second order normalized Hamiltonian (13) using Lie transform method described in Coppola and Rand [5, 7], Jorba [15], Celletti [3], Kishor and Kushvah [19]. As, in the paper of Coppola and Rand [7] as well as in the book of Celletti [3], higher order normalized Hamiltonian is

$$\begin{aligned} K= & {} K_2+K_3+K_4+\cdots +K_n+\cdots , \end{aligned}$$
(46)

where

$$\begin{aligned} K_n= & {} \sum {K_{ijkl}X^iY^j{P_X}^k{P_Y}^l} \end{aligned}$$
(47)

such that \(i+j+k+l=n.\) Quadratic part of K is \(K_2=H_2\), whereas \(K_n\) through the nth step of Lie transform is given as

$$\begin{aligned} K_n= & {} \frac{1}{n}\{H_2,\,G_n\}+(\text {known terms}), \end{aligned}$$
(48)

where Lie bracket of normalized quadratic Hamiltonian \(H_2\) and generating function \(G_n\) is defined as

$$\begin{aligned} \{H_2,\,G_n\}=\frac{\partial H_2}{\partial X}\frac{\partial G_n}{\partial P_X}- \frac{\partial H_2}{\partial P_X}\frac{\partial G_n}{\partial X}+ \frac{\partial H_2}{\partial Y}\frac{\partial G_n}{\partial P_Y}-\frac{\partial H_2}{\partial P_Y}\frac{\partial G_n}{\partial Y}. \end{aligned}$$
(49)

Using \(H_2\) from Eq. (13), it reduces to

$$\begin{aligned} \{H_2,\,G_n\}=\mathbf {i}\omega _1\left( P_X\frac{\partial G_n}{\partial P_X}-X\frac{\partial G_n}{\partial X}\right) +\mathbf {i}\omega _2\left( P_Y\frac{\partial G_n}{\partial P_Y}-Y\frac{\partial G_n}{\partial Y}\right) . \end{aligned}$$
(50)

The choice of generating function \(G_n\) is such that the above partial differential operator on \(G_n\) remove large possible number of terms from the expression of \(K_n\). As, each terms of the \(K_n\) is of the form \(\alpha X^iY^j{P_X}^k{P_Y}^l\), where \(\alpha \) is constant, we can assume terms in \(G_n\) of the form \(\beta X^iY^j{P_X}^k{P_Y}^l\), where constant \(\beta \) is to be determined. Therefore, we obtain that

$$\begin{aligned} \frac{\{H_2,\,G_n\}}{n}=\frac{\mathbf {i}\beta }{n}\left[ (k-i)\omega _1-(l-j)\right] X^iY^j{P_X}^k{P_Y}^l, \end{aligned}$$
(51)

and hence,

$$\begin{aligned} \beta =\frac{\mathbf {i}\alpha }{\left[ (k-i)\omega _1-(l-j)\right] },\,\,i +j+k+l=n. \end{aligned}$$
(52)

This shows that even in the non-resonance case, the term of the form \(X^iY^j{P_X}^i{P_Y}^j\) in \(K_n\) can not be deleted because of vanishing denominator in (52) at \(i=k,\,j=l\), whereas in the resonance case some additional non-removable terms occur while solving the generating function \(G_n\). Hence, in non-resonance case, the Hamiltonian of the present problem can be written in the form of (46), in which

$$\begin{aligned} K_2= & {} i\omega _1XP_X -i\omega _2YP_Y, \end{aligned}$$
(53)
$$\begin{aligned} K_3= & {} 0, \end{aligned}$$
(54)
$$\begin{aligned} K_4= & {} \frac{AX^2P_{X}^2+BXP_XYP_Y+CY^2P_{Y}^2}{2}, \end{aligned}$$
(55)

where \(A=2K_{2020},\,B=2K_{1111},\) and \(C=2K_{0202}\). Using action variables \(I_1=iXP_X\) and \(I_2=iYP_Y\) in Eqs. (53–55), we get

$$\begin{aligned} K_2= & {} \omega _1I_1 -\omega _2I_2, \end{aligned}$$
(56)
$$\begin{aligned} K_3= & {} 0,, \end{aligned}$$
(57)
$$\begin{aligned} K_4= & {} -\left( AI_{1}^2+BI_I1_2+CI_{2}^2\right) . \end{aligned}$$
(58)

Thus, normalized Hamiltonian up to fourth order is

$$\begin{aligned} K(I_1,\,I_2)= & {} K_2+K_3+K_4\nonumber \\= & {} \omega _1I_1 -\omega _2I_2-K_{2020}I_{1}^2+K_{1111}I_I1_2+K_{0202}I_{2}^2, \end{aligned}$$
(59)

which agree with that of Deprit and Deprit-Bartholome [9], Kushvah et al. [20], Kishor and Kushvah [19].

Form Eq. (59), it is clear that fourth order normalized Hamiltonian is the function of only action variables \(I_1,\, I_2\), which shows that these are in Birkhoff normal form. The coefficients \(K_{ijkl}\) used in the Eq. (55 or 58) can be written into 5 parts such as \(K_{ijkl1}\), \(K_{ijkl2}\), \(K_{ijkl3}\), \(K_{ijkl4}\) and \(K_{ijkl5}\) for \(i,\,j,\,k,\,l=0,\,1,\,2,\,3,\,4\) such that \(i+j+k+l=4\). These coefficients corresponds to the term of classical part, terms with P–R drags \(W_1\) and \(W_2\) of first and second primary, radiation pressures \(\epsilon _1=1-q_1\) and \(\epsilon _2=1-q_2\) of first and second primary, respectively. In the absence of perturbing parameters i.e. for \(W_1=W_1=\epsilon _1=\epsilon _2=0\), \(K_{ijkl}=K_{ijkl1}\). Therefore, \(K_{2020},\,K_{1111}\) and \(K_{0202}\) become

$$\begin{aligned} K_{2020}= & {} K_{20201}+K_{20202}+K_{20203}+K_{20204}+K_{20205}, \end{aligned}$$
(60)
$$\begin{aligned} K_{1111}= & {} K_{11111}+K_{11112}+K_{11113}+K_{11114}+K_{11115}, \end{aligned}$$
(61)
$$\begin{aligned} K_{0202}= & {} K_{02021}+K_{02022}+K_{02023}+K_{02024}+K_{02025}. \end{aligned}$$
(62)

The algebraic expressions of above 15 coefficients on right hand sides of Eqs. (60–62) are too complicated and huge to be placed here hence, we avoid to present in the paper. These are utilized to compute the determinant \(D_{4}=K_4(\omega _2,\,\omega _1)\) for applying the Arnold–Moser theorem. For the simplicity, \(D_4\) is expressed as

$$\begin{aligned} D_4=\left( \frac{A_1}{B_1}\right) +\left( \frac{A_2}{B_2}\right) W_1+\left( \frac{A_3}{B_3}\right) W_2+\left( \frac{A_4}{B_4}\right) \epsilon _1+\left( \frac{A_5}{B_5}\right) \epsilon _2, \end{aligned}$$
(63)

where \(A_i,\, B_i,\, i=1,2,3,4,5\) are numerator and denominator of the coefficients, which correspond to classical part, P–R drags \(W_1\) and \(W_2\) of the primaries, radiation pressure \(\epsilon _1=1-q_1\) and \(\epsilon _2=1-q_2\) of the primaries, respectively. On simplification, we found that

Fig. 2
figure 2

Zero \((\mu _0)\) of the determinant \(D_4\) within the stability range \(0<\mu <\mu _c\) at: a\(W_1=W_2=\epsilon _1=\epsilon _2=0\) (classical case); b\(W_1=0.015,\,W_2=0.005,\,\epsilon _1=0.003,\,\epsilon _2=0.03\) (perturbed case)

Full size image
Fig. 3
figure 3

Zero \((\mu _0)\) of the determinant \(D_4\) within the stability range \(0<\mu <\mu _c\) at: a\(W_1=0.015,\,W_2=\epsilon _1=\epsilon _2=0\) (only in presence of P–R drag of first primary); b zoom of specified region of a

Full size image
Fig. 4
figure 4

Zero \((\mu _0)\) of the determinant \(D_4\) within the stability range \(0<\mu <\mu _c\) at: a\(W_2=0.005,\,W_1=\epsilon _1=\epsilon _2=0\) (only in presence of P–R drag of second primary); b zoom of specified region of a

Full size image
Fig. 5
figure 5

Zero \((\mu _0)\) of the determinant \(D_4\) within the stability range \(0<\mu <\mu _c\) at: \(\epsilon _1=1-q_1=0.003,\,W_1=W_2=\epsilon _2=0\) (only in presence of radiation pressure of first primary)

Full size image
Fig. 6
figure 6

Zero \((\mu _0)\) of the determinant \(D_4\) within the stability range \(0<\mu <\mu _c\) at \(\epsilon _2=1-q_2=0.03,\,W_1=W_2=\epsilon _1=0\) (only in presence of radiation pressure of second primary)

Full size image
$$\begin{aligned} A_1= & {} -35+541\omega _{1}^2\omega _{2}^2-644\omega _{1}^4\omega _{2}^4, \end{aligned}$$
(64)
$$\begin{aligned} A_2= & {} 26244\left( 2262-653b\right) -27\left( 5292162-4787719 b\right) \omega _{1}^2\omega _{2}^2\nonumber \\&-2\left( 402982614-10430203 b\right) \omega _{1}^4\omega _{2}^4+32\left( 12457908-1490819b\right) \omega _{1}^6\omega _{2}^6\nonumber \\&+1024\left( 67581+1634b\right) \omega _{1}^8\omega _{2}^8, \end{aligned}$$
(65)
$$\begin{aligned} A_3= & {} -78732\left( 416-241\sqrt{3} b\right) -27\left( 3181248+4414649sqrt{3} b\right) \omega _{1}^2\omega _{2}^2\nonumber \\&-6\left( 72610776+10390609\sqrt{3} b\right) \omega _{1}^4\omega _{2}^4+32\left( 6808752-212191\sqrt{3} b\right) \omega _{1}^6\omega _{2}^6\nonumber \\&+1024\left( 36828+997\sqrt{3} b\right) \omega _{1}^8\omega _{2}^8, \end{aligned}$$
(66)
$$\begin{aligned} A_4= & {} 8748\left( 195\sqrt{3}-584 b\right) -27\left( 465795\sqrt{3}-1556744 b\right) \omega _{1}^2\omega _{2}^2\nonumber \\&+2\left( 10722915\sqrt{3}-11609036 b\right) \omega _{1}^4\omega _{2}^4+32\left( 200970\sqrt{3}-103079 b\right) \omega _{1}^6\omega _{2}^6\nonumber \\&-512\left( 2565\sqrt{3}-3217 b\right) \omega _{1}^8\omega _{2}^8, \end{aligned}$$
(67)
$$\begin{aligned} A_5= & {} -8748\left( 507\sqrt{3}-688 b\right) +27\left( 315819\sqrt{3}-1533728 b\right) \omega _{1}^2\omega _{2}^2\nonumber \\&+2\left( 3085838\sqrt{3}+759212 b\right) \omega _{1}^4\omega _{2}^4-32\left( 941526\sqrt{3}-283835 b\right) \omega _{1}^6\omega _{2}^6\nonumber \\&-512\left( 10251\sqrt{3}+877 b\right) \omega _{1}^8\omega _{2}^8, \end{aligned}$$
(68)
$$\begin{aligned} B_1= & {} 8\left( 1-4\omega _{1}^2\omega _{2}^2\right) \left( 4-25\omega _{1}^2\omega _{2}^2\right) , \end{aligned}$$
(69)
$$\begin{aligned} B_2= & {} 864ab, \end{aligned}$$
(70)
$$\begin{aligned} B_3= & {} B_2, \end{aligned}$$
(71)
$$\begin{aligned} B_4= & {} 1152ab, \end{aligned}$$
(72)
$$\begin{aligned} B_5= & {} 576ab, \end{aligned}$$
(73)
$$\begin{aligned} a= & {} \left[ \omega _{1}^2\omega _{2}^2\left( 1-4\omega _{1}^2\omega _{2}^2\right) \right. \nonumber \\&\left. \left( 4-25\omega _{1}^2\omega _{2}^2\right) \left( 117+16\omega _{1}^2\omega _{2}^2 \right) \right] , \end{aligned}$$
(74)
$$\begin{aligned} b= & {} \sqrt{\left( 27-16\omega _{1}^2\omega _{2}^2\right) }. \end{aligned}$$
(75)

In the absence of perturbing parameters,

$$\begin{aligned} D_4= & {} \frac{-35+541\omega _{1}^2\omega _{2}^2-644\omega _{1}^4\omega _{2}^4}{8 \left( 1-4\omega _{1}^2\omega _{2}^2\right) \left( 4-25\omega _{1}^2\omega _{2}^2\right) }, \end{aligned}$$
(76)

which agree with the classical result [9, 19, 20, 25]. In order to analyze the nonlinear stability of triangular equilibrium points in non-resonance case using Arnold–Moser theorem, we plot the determinant \(D_4\) with respect to the mass parameter \(\mu \) to insure the value of \(D_4=K_4(\omega _2,\omega _1)\). From Figs. 2, 3, 4, 5 and 6, it is clear that within the linear stability range \(0<\mu <\mu _c\), there exists one value of mass parameter \(\mu =\mu _0\), called the zero of \(D_4\), at which \(D_4\) vanish in each case. Thus, Arnold–Moser theorem fails, which insure that in non-resonance case, triangular equilibrium points of the problem under analysis are unstable in nonlinear sense within the linear stability range \(0<\mu <\mu _c\). To see the effect of perturbing parameters, we have computed values of the zero \((\mu _0)\) of \(D_4\) and critical mass ratio \((\mu _c)\) at different values of perturbing parameters \(W_1,\,W_2,\,\epsilon _1,\,\epsilon _2\) and results are placed in Table 2. From Table 2, it is noticed that on increase in the values of \(W_1,\,W_1,\,\epsilon _1\), value of critical mass \(\mu _c\) increases but the value of \(\mu _0\) is nonzero in each case. On the other hand, on increase in the value of \(\epsilon _2\), \(\mu _c\) decreases with nonzero \(\mu _0\). Thus, from the Figs. 2, 3, 4, 5 and 6 as well as from the Table 2, it is clear that radiation pressure and P–R drag of both the primaries affect the linear stability range of the problem significantly. The nonzero value of the zero \((\mu _0)\) of the determinant \(D_4\) in the Arnold–Moser theorem under non-resonance case, insure the instability of triangular equilibrium points, within the range of stability \(0<\mu <\mu _c\).

Table 2 Zero \((\mu _0)\) of \(D_4\) and critical mass ratio \(\mu _c\) at different values of perturbing parameters
Full size table

6 Conclusions

We have considered the photogravitational restricted three body problem in the presence of radiation pressure force and P–R drag of both the massive bodies, which are radiating in nature. Analysis of nonlinear stability of the triangular equilibrium points is performed in non-resonance case using Arnold–Moser theorem under the influence of four perturbing parameters in the form of P–R drags \(W_1, \, W_2\) and mass reduction factors \(q_1,\,q_2\), of both the primaries. First, we have normalized the Hamiltonian of the problem up to order four using Lie transform method and then Birkhoff normal form of the Hamiltonian constructed, which is necessary to apply the Arnold–Moser theorem in non-resonance case. The determinant \(D_4\) of the Arnold–Moser theorem is computed analytically under the consideration of only linear order terms of perturbing parameters, which agree with that of Deprit and Deprit-Bartholome [9], Meyer and Schmidt [25], Kushvah et al. [20], Kishor and Kushvah [19] in the absence of perturbing parameters. To apply the Arnold–Moser theorem in non-resonance case, we have plotted the determinant \(D_4\) with respect to the mass parameter \(\mu \) within the stability range \(0<\mu <\mu _c\). It is observed that in presence as well as in absence of perturbing parameters, there exist a nonzero value of \(\mu =\mu _0\) at which \(D_4\) vanish (Figs. 2, 3, 4, 5, 6), which insure that triangular equilibrium points are unstable in nonlinear sense. The effect of perturbing parameters are also analyzed and it is found that on increasing the values of \(W_1,\,W_1,\,\epsilon _1\), critical mass ratio \(\mu _c\) increases, with the existence of nonzero \(\mu _0\) in each case, whereas on increasing the value of \(\epsilon _2\), \(\mu _c\) decreases with the existence of nonzero \(\mu _0\) (Fig. 1; Table 2). A similar trend is also seen in case of \(\mu _{c1}\) and \(\mu _{c2}\) (Table 1). Thus, we conclude that due to radiation pressure and P–R drag of both the primaries, the linear stability range of the problem get changed, significantly. Also, due to existence of nonzero value of the zero \((\mu _0)\) of the determinant \(D_4\) in the Arnold–Moser theorem under non-resonance case, within the range of stability \(0<\mu <\mu _c\), triangular equilibrium points are unstable in nonlinear sense. Present analysis is limited up to first order terms of the perturbing parameter, which may be extended to higher order inclusion of the terms. The results obtained can help to analyze the more generalized problem under the influence of other perturbations such as albedo, solar wind drag, Stokes drag etc.