1 Introduction

The restricted problem of three bodies is a well known problem studied by many mathematicians. The main aim of this problem is to study the behavior of the infinitesimal mass moving in the plane of motion of the primaries under the various effects such as gravitational effect, radiation effect, oblateness effect, solar wind effect, Stokes drag effect, Poynting Robertson drag effect etc. In classical case, gravitational effect of the primaries on the infinitesimal mass is taken into account and there exist three collinear and two non-collinear libration points. The collinear libration points are unstable in interval \(0 \le \mu \le 1/2\) while non-collinear libration points are stable for a critical value of mass parameter \(\mu \le \mu_{c} = 0.03852\ldots{}\) (Szebehely 1967). In contrast to the classical case, by including Stokes drag the collinear libration points does not exist but the non-collinear libration points do exist and are unstable for all values of \(\mu\).

Many authors have been studied this problem by taking one or both the primaries as an oblate body including radiation pressure. Subbarao and Sharma (1975) has investigated the non-collinear libration points in circular restricted three body problem considering bigger primary as an oblate spheroid and found that the non-collinear libration points forming nearly equilateral triangles with the primaries. Murray (1994) has discussed the dynamical effect of general drag in the planar circular restricted three body problem and found that \(L_{4}\) and \(L_{5}\) are asymptotically stable with this kind of dissipation. Sharma et al. (2001) have performed an analysis on the existence of libration points when both the primaries are triaxial rigid bodies. They have shown that there exist five libration points, two triangular and three collinear. Shu et al. (2004) have discussed the linear stability of the equilibrium points in the Robes problem under the perturbation of a drag force. They have derived the linearly stable region of the equilibrium point in the perturbed Robes problem with the drag given by Hallan et al., and improved as well the results obtained by Giordano et al. Raheem and Singh (2006) have studied the existence of the stability of libration points under the effects of perturbation in coriolis and centrifugal forces, oblateness and radiation pressure. They have found that the collinear points remain unstable while the triangular points are stable for \(0 \le \mu < \mu_{c}\) and unstable for \(\mu_{c} \le \mu \le 1/2\), where \(\mu_{c}\) is the critical mass parameter depends upon the coriolis force, centrifugal force, oblateness and radiation pressure of the primaries. Aggarwal et al. (2006) have investigated the non-linear stability of the triangular libration point \(L_{4}\) of the restricted three body problem under the presence of the third and fourth order resonances, when the bigger primary is an oblate body and the smaller a triaxial body and both are source of radiation. It has found that \(L_{4}\) is always unstable in the third resonance case and stable or unstable in the fourth order resonance case depending upon the values of the different parameters. Kushvah et al. (2007) have discussed the non-linear stability in the generalized restricted three body problem with Poynting Robertson drag considering smaller primary as an oblate body and bigger one as radiating. They have proved that the triangular points are stable in non-linear sense. Abouelmagd (2013) has studied the existence of triangular points and their linear stability when the primaries are oblate spheroid and sources of radiation considering the effect of oblateness up to \(10^{-6}\) of main terms in the restricted three body problem. He also proved that the triangular points are stable for \(0 \le \mu \le \mu_{c}\) and unstable for \(\mu_{c} \le \mu \le 1/2\), where \(\mu_{c}\) is the critical mass value depending on terms which involve parameters that characterize the oblateness and radiation repulsive forces.

Furthermore, Aggarwal and Kaur (2014) have analyzed the equilibrium solutions and the linear stability of \(m_{3}\) and \(m_{4}\) by taking one of the primaries as an oblate spheroid. They have found that the two collinear libration points are unstable and also found that in this particular case there are no non-collinear equilibrium solutions of the system. Lhotka and Celletti (2015) have investigated the stability of the Lagrangian equilibrium points \(L_{4}\) and \(L_{5}\) in the framework of the spatial elliptical restricted three body problem subject to the radial component of Poynting Robertson drag. They have used averaging theory (i.e. average over the mean anomaly of the perturbing planet) to discuss the temporary stability of particles displaying tadpole motion. Pal and Kushvah (2015) have determined the effect of radiation pressure, Poynting Robertson drag and solar wind drag on the sun- (earth-moon) restricted three body problem considering sun as a larger primary and the earth + moon as a smaller primary and found that the collinear points deviate from the axis joining the primaries, but the triangular points remain unchanged. They have also found that triangular points are unstable because of the drag forces. Jain and Aggarwal (2015) have performed an analysis in the restricted three body problem with Stokes drag effect. By taking both primaries \(m_{1}\), \(m_{2}\) as the point masses, we found two non-collinear stationary solutions which are linearly unstable.

We have extended the study of Jain and Aggarwal (2015) to the restricted three body problem when one of the primaries is an oblate spheroid. In this paper we are considering the smaller primary as an oblate spheroid and the bigger one as a point mass. In the present paper, our aim is to study the combined effect of stokes drag and oblateness on the stability of non-collinear libration points \(L_{4}\) and \(L_{5}\) linearly. There are five sections in this paper. In Sect. 2, the equations of motion of the infinitesimal mass \(m_{3}\) have been determined. In Sect. 3, location of the non-collinear libration points have been investigated. In Sect. 4, we have checked the stability of the non-collinear libration points. In the last Sect. 5, the conclusion is drawn.

2 Equations of motion

Suppose \(m_{1}\) and \(m_{2}\) are the primaries revolving with angular velocity \(n\) in circular orbits about their center of mass \(O\), an infinitesimal mass \(m_{3}\) is moving in the plane of motion of \(m_{1}\) and \(m_{2}\). The line joining \(m_{1}\) and \(m_{2}\) is taken as X-axis and ‘\(O\)’ their center of mass as origin and the line passing through \(O\) and perpendicular to \(OX\) and lying in the plane of motion of \(m_{1}\) and \(m_{2}\) is the \(Y\)-axis. We consider a synodic system of coordinates \(O\) (xyz); initially coincident with the inertial system \(O\) (XYZ), rotating with the angular velocity n about \(Z\)-axis; (the \(z\)-axis is coincident with \(Z\)-axis) (Fig. 1).

Fig. 1
figure 1

Configuration of the restricted three body problem with Stokes Drag \(\vec{S}\)

The equations of motion of the infinitesimal mass \(m_{3}\) in the synodic coordinate system and dimensionless variables when bigger primary is a point mass and smaller one is an oblate spheroid are

$$ \ddot{x} - 2n\dot{y} = \Omega_{x} - k \bigl(\dot{x} - y + \alpha S'_{y}\bigr), $$
(1)

and

$$ \ddot{y} + 2n\dot{x} = \Omega_{y} - k \bigl(\dot{y} + x - \alpha S'_{x}\bigr). $$
(2)

where

$$\begin{aligned} &\Omega_{x} = n^{2}x - (1 - \mu )\frac{(x - \mu )}{r_{{1}}^{3}} - \mu \frac{(x + 1 - \mu )}{r_{2}^{3}} \biggl( 1 + \frac{3A}{2r_{2}^{2}} \biggr), \\ &\Omega_{y} = n^{2}y - \frac{(1 - \mu )}{r_{{1}}^{3}}y - \frac{\mu}{ r_{{2}}^{3}}y \biggl( 1 + \frac{3A}{2r_{{2}}^{2}} \biggr), \\ &n = 1 + \frac{3}{4}A\ \mbox{is the mean motion of the primaries}, \\ &A = \frac{a^{2} - c^{2}}{5}\ \mbox{is the oblateness factor}, \end{aligned}$$
$$\begin{aligned} &r_{1}^{2} = (x - \mu )^{2} + y^{2}, \end{aligned}$$
(3)
$$\begin{aligned} &r_{2}^{2} = (x + 1 - \mu )^{2} + y^{2}, \end{aligned}$$
(4)
$$\begin{aligned} &\mu = \frac{m_{2}}{m_{1} + m_{2}} \le \frac{1}{2} \quad \Rightarrow\quad m_{1} = 1 - \mu; m_{2} = \mu, \\ & \begin{aligned}[t] \vec{S}&= \mbox{Stokes drags Force acting on}\ m_{3}\ \mbox{due to}\ m_{1}\ \mbox{along}\\ &\quad {} m_{1}\ m_{3}. \end{aligned} \end{aligned}$$

The components of Stokes drag along the synodic axes \((x, y)\) are \(S_{x} = k(\dot{x} - y) + \alpha S'_{y}\) and \(S_{y} = k(\dot{y} + x) - \alpha S'_{x}\), where \(k \in (0, 1)\) is the dissipative constant, depending on several physical parameters like the viscosity of the gas, the radius and mass of the particle.

\(S' = S'(r) = r^{\frac{ - 3}{2}}\), is the keplerian angular velocity at distance \(r = \sqrt{x^{2} + y^{2}}\) from the origin of the synodic frame and \(\alpha \in (0, 1)\) is the ratio between the gas and keplerian velocities.

$$\begin{aligned} &\vec{r} = \overline{OP} = xi + yj,\\ &\vec{\omega} = nK =\mbox{Angular velocity of the axes}\ O(x y)= \mbox{constant}. \end{aligned}$$

The Stokes drag effect is of the order of \(k = 10^{ - 5}\), \(\alpha = 0.05\) (generally \(k \in (0, 1)\) and \(\alpha \in (0, 1)\) as stated above).

3 Non-collinear libration points

The non-collinear libration points are the solution of the equations

$$\begin{aligned} &n^{2}x - (1 - \mu )\frac{(x - \mu )}{r_{{1}}^{3}} - \mu \frac{(x + 1 - \mu )}{r_{{2}}^{3}} \biggl( 1 + \frac{3A}{2r_{2}^{2}} \biggr) \\ &\quad {} + k \biggl( y + \frac{3}{2}\alpha \bigl(x^{2} + y^{2}\bigr)^{\frac{ - 7}{4}}y \biggr) = 0, \end{aligned}$$
(5)

and

$$\begin{aligned} &n^{2}y - \frac{(1 - \mu )}{r_{{1}}^{3}}y - \frac{\mu}{ r_{{2}}^{3}}y \biggl( 1 + \frac{3A}{2r_{{2}}^{2}} \biggr) \\ &\quad {}- k \biggl( x - \frac{3}{2}\alpha \bigl(x^{2} + y^{2}\bigr)^{\frac{ - 7}{4}}x \biggr) = 0. \end{aligned}$$
(6)

In the above equations, if we put \(k = 0\), the obtained results are agreed with Khanna and Bhatnagar (1999) i.e.

$$n^{2}x - (1 - \mu )\frac{(x - \mu )}{r_{{1}}^{3}} - \mu \frac{(x + 1 - \mu )}{r_{{2}}^{3}} \biggl( 1 + \frac{3A}{2r_{{2}}^{2}} \biggr) = 0, $$

and

$$n^{2}y - \frac{(1 - \mu )}{r_{{1}}^{3}}y - \frac{\mu}{r_{{2}}^{3}}y \biggl( 1 + \frac{3A}{2r_{{2}}^{2}} \biggr) = 0. $$

Due to the presence of the Stokes drag force, it is clear from Eqs. (5) and (6) that collinear libration solution does not exist, so we restrict our analysis to these points. The location of the non-collinear libration points when smaller primary is an oblate spheroid are given by (Khanna and Bhatnagar 1999)

$$\begin{aligned} &x_{0} = \mu - \frac{1}{2}(1 - A), \\ & y_{0} = \pm \frac{\sqrt{3}}{2} \biggl( 1 - \frac{A}{3} \biggr). \end{aligned}$$

Now, we suppose that the solution of the Eqs. (5) and (6) when \(k \ne 0\)and \(y \ne 0\) are given by

$$\bar{x} = x_{0} + \pi_{1},\qquad \bar{y} = y_{0} + \pi_{2},\quad \pi_{1}, \pi_{2} < < 1 $$

On substituting the values of (\(\bar{x},\bar{y}\)) in Eqs. (5) and (6), and applying Taylor’s series and considering only linear terms in \(\pi_{1}\) and \(\pi_{2}\), we get

$$\begin{aligned} &\pi_{1} \biggl[ 1 + (1 - \mu )\frac{3(x_{0} - \mu )^{2}}{ \{ (r_{1})^{2} \}^{\frac{5}{2}}} - \frac{1}{ \{ (r_{1})^{2} \}^{\frac{3}{2}}} \\ &\quad {} + \mu \frac{3(x_{0} + 1 - \mu )^{2}}{ \{ (r_{2})^{2} \}^{\frac{5}{2}}} - \frac{1}{ \{ (r_{2})^{2} \}^{\frac{3}{2}}} \biggr] \\ &\quad {} + \pi_{2} \biggl[ (1 - \mu )\frac{3(x_{0} - \mu )y_{0}}{ \{ (r_{1})^{2} \}^{\frac{5}{2}}}+ \mu \frac{3(x_{0} + 1 - \mu )y_{0}}{ \{ (r_{2})^{2} \}^{\frac{5}{2}}} \biggr] \\ &\quad {} + A \biggl( \frac{3y_{0}(x_{0} + 1 - \mu )\mu}{ \{ (r_{2})^{2} \}^{\frac{7}{2}}}\pi_{2} + \frac{9}{2} \frac{(x_{0} + 1 - \mu )^{2}\mu}{ \{ (r_{2})^{2} \}^{\frac{7}{2}}}\pi_{1} \\ &\quad {} + \frac{3\mu (x_{0} + 1 - \mu )^{2}}{ \{ (r_{2})^{2} \}^{\frac{7}{2}}}\pi_{1} + \frac{3}{2}\pi_{1} + \frac{9}{2}\frac{\mu y_{0}(x_{0} + 1 - \mu )}{ \{ (r_{2})^{2} \}^{\frac{7}{2}}} \pi_{2} \\ &\quad {}- \frac{3}{2}\frac{\mu (x_{0} + 1 - \mu )}{ \{ (r_{2})^{2} \}^{\frac{7}{2}}} - \frac{3\mu}{2 \{ (r_{2})^{2} \}^{\frac{5}{2}}} \pi_{1} \biggr) \\ &\quad {} + k \biggl[y_{0} + \frac{3}{2}\alpha \bigl(x_{0}^{2} + y_{0}^{2}\bigr)^{\frac{ - 7}{4}}y_{0}\biggr] = 0 \end{aligned}$$
(7)

and

$$\begin{aligned} &\pi_{2} \biggl[ 1 + (1 - \mu )\frac{3y_{0}^{2}}{ \{ (r_{1})^{2} \}^{\frac{5}{2}}} - \frac{1}{ \{ (r_{1})^{2} \}^{\frac{3}{2}}} + \mu \frac{3y_{0}^{2}}{ \{ (r_{2})^{2} \}^{\frac{5}{2}}} \\ &\quad {}- \frac{1}{ \{ (r_{2})^{2} \}^{\frac{3}{2}}} \biggr] + \pi_{1} \biggl[ (1 - \mu )\frac{3(x_{0} - \mu )y_{0}}{ \{ (r_{1})^{2} \}^{\frac{5}{2}}} \\ &\quad {}+ \mu \frac{3(x_{0} + 1 - \mu )y_{0}}{ \{ (r_{2})^{2} \}^{\frac{5}{2}}} \biggr] + A \biggl( \frac{3y_{0}(x_{0} + 1 - \mu )\mu}{ \{ (r_{2})^{2} \}^{\frac{7}{2}}}\pi_{1} \\ &\quad {}+ \frac{9}{2} \frac{y_{0}(x_{0} + 1 - \mu )^{2}\mu}{ \{ (r_{2})^{2} \}^{\frac{7}{2}}}\pi_{1} + \frac{3\mu y_{0}^{2}}{ \{ (r_{2})^{2} \}^{\frac{7}{2}}}\pi_{2} + \frac{3}{2}\pi_{2} \\ &\quad {}+ \frac{9}{2}\frac{\mu y_{0}^{2}}{ \{ (r_{2})^{2} \}^{\frac{7}{2}}} \pi_{2} - \frac{3}{2}\frac{\mu}{ \{ (r_{2})^{2} \}^{\frac{5}{2}}}\pi_{2} - \frac{3y_{0}\mu}{2 \{ (r_{2})^{2} \}^{\frac{7}{2}}} \biggr) \\ &\quad {} - k \biggl[x_{0} + \frac{3}{2}\alpha \bigl(x_{0}^{2} + y_{0}^{2}\bigr)^{\frac{ - 7}{4}}x_{0}\biggr] = 0. \end{aligned}$$
(8)

where

$$\begin{aligned} &r_{{1}}^{2} = (x_{0} - \mu )^{2} + y_{0}^{2}, \\ & r_{{2}}^{2} = (x_{0} + 1 - \mu )^{2} + y_{0}^{2}, \end{aligned}$$

Since \(x_{0} = \mu - \frac{1}{2}(1 - A)\) and \(y_{0} = \pm \frac{\sqrt{3}}{ 2} ( 1 - \frac{A}{3} )\), therefore on solving Eqs. (7) and (8), we have

$$\begin{aligned} &\pi_{1} = - \frac{1}{2\sqrt{3} A}k - \frac{\sqrt{3} k \alpha}{2A}, \\ & \pi_{2} = \frac{k}{6A} + \frac{\alpha k}{2A} + \frac{\mu k}{6A}. \end{aligned}$$

Hence, the location of the non-collinear libration points \(L_{4}\) and \(L_{5}\) are given by

$$ \begin{aligned}[c] &\bar{x} = \mu - \frac{1}{2}(1 - A) - \frac{k}{2\sqrt{3} A} + \frac{3\sqrt{3} \alpha}{4A^{2}} + \frac{\sqrt{3}}{4A^{2}}, \\ &\bar{y} = \pm \frac{\sqrt{3}}{2} \biggl( 1 - \frac{A}{3} \biggr) + \frac{1}{6A}\mu k + \frac{\alpha k}{2A} + \frac{k}{6A}. \end{aligned} $$
(9)

4 Stability of \(L_{4,5}\)

The variational equations can be written by substituting \(x = \bar{x} + \xi\) and \(y = \bar{y} + \eta\) in the equations of motion (1) and (2), where (\(\bar{x},\bar{y}\)) are the coordinates of the non-collinear libration points.

Therefore, expanding \(f(\bar{x},\bar{y})\) and \(g(\bar{x},\bar{y})\) by Taylors Theorem, we get

$$\begin{aligned} \ddot{\xi} - 2 \dot{\eta} &= \Omega_{x}(\bar{x},\bar{y}) + \xi \biggl[ n^{2} - \frac{(1 - \mu )}{(\bar{r}_{1})^{3}} + \frac{3 (1 - \mu )(\bar{x} - \mu )^{2}}{(\bar{r}_{1})^{5}} \\ &\quad + \frac{3 \mu (\bar{x} + 1 - \mu )^{2}}{(\bar{r}_{2})^{5}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) - \frac{\mu}{ (\bar{r}_{2})^{3}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) \\ &\quad + \frac{3 \mu A(\bar{x} + 1 - \mu )^{2}}{(\bar{r}_{2})^{7}} - k - \frac{21 \bar{x} \bar{y} \alpha}{4}\bigl(\bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}k \biggr] \\ &\quad + \eta \biggl[ \frac{3 \bar{y}\mu (\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{5}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) \\ &\quad + \frac{3\bar{y} (1 - \mu )(\bar{x} - \mu )}{(\bar{r}_{1})^{5}} + \frac{3 \bar{y}\mu (\bar{x} + 1 - \mu )A}{(\bar{r}_{2})^{7}} + k \\ &\quad + \frac{3}{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 7}{4}}k - \frac{21}{4}\bar{y}^{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}k \biggr], \end{aligned}$$
(10)
$$\begin{aligned} \ddot{\eta} + 2 \dot{\xi} &= \Omega_{y}(\bar{x},\bar{y}) + \xi \biggl[ \frac{3 \bar{y} \mu (\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{5}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) \\ &\quad+ \frac{3\bar{y} (1 - \mu )(\bar{x} - \mu )}{(\bar{r}_{1})^{5}} + \frac{3 \bar{y} \mu (\bar{x} + 1 - \mu )A}{(\bar{r}_{2})^{7}} - k \\ &\quad- \frac{3}{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 7}{4}}k + \frac{21}{4}\bar{x}^{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}k \biggr] \\ &\quad + \eta \biggl[ n^{2} - \frac{(1 - \mu )}{(\bar{r}_{1})^{3}} - \frac{\mu}{(\bar{r}_{2})^{3}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) \\ &\quad + \bar{y}^{2} \biggl\{ \frac{3 (1 - \mu )}{(\bar{r}_{1})^{5}} + \frac{3 \mu}{ (\bar{r}_{2})^{5}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) + \frac{3 \mu A}{\bar{r}_{{2}}^{7}} \biggr\} \\ &\quad - k + \frac{21\bar{x} \bar{y} \alpha}{4}\bigl(\bar{x}^{2} + \bar{y}^{2} \bigr)^{\frac{ - 11}{4}}k \biggr]. \end{aligned}$$
(11)

Suppose the trial solution of Eqs. (10) and (11) is

$$\xi = \xi_{0}e^{\lambda t},\quad \eta = \eta_{0}e^{\lambda t} $$

where \(\xi_{0}\) and \(\eta_{0}\) are constants and \(\lambda\) is a complex constant. Then we have

$$\begin{aligned} &\lambda^{2} \xi_{0} e^{\lambda t} - 2 \lambda \eta_{0} e^{\lambda t} \\ &\quad = \xi_{0} e^{\lambda t} \biggl[ n^{2} - \frac{(1 - \mu )}{(\bar{r}_{1})^{3}} + \frac{3 (1 - \mu )(\bar{x} - \mu )^{2}}{(\bar{r}_{1})^{5}} \\ &\qquad{} + \frac{3 \mu (\bar{x} + 1 - \mu )^{2}}{(\bar{r}_{2})^{5}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) - \frac{\mu}{ (\bar{r}_{2})^{3}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) \\ &\qquad{} + \frac{3 \mu A(\bar{x} + 1 - \mu )^{2}}{(\bar{r}_{2})^{7}} + \lambda k - \frac{21 \bar{x} \bar{y} \alpha}{4}\bigl(\bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}k \biggr] \\ &\qquad{} + \eta_{0} e^{\lambda t} \biggl[ \frac{3 \bar{y}\mu (\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{5}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) \\ &\qquad{}+ \frac{3\bar{y} (1 - \mu )(\bar{x} - \mu )}{(\bar{r}_{1})^{5}} + \frac{3 \bar{y}\mu (\bar{x} + 1 - \mu )A}{(\bar{r}_{2})^{7}} + \lambda k \\ &\qquad{} + \frac{3}{2}\alpha \bigl( \bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 7}{4}}k - \frac{21}{4}\bar{y}^{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}k \biggr], \end{aligned}$$
(12)
$$\begin{aligned} & \lambda^{2} \eta_{0} e^{\lambda t} + 2 \lambda \xi_{0} e^{\lambda t} \\ &\quad = \xi_{0} e^{\lambda t} \biggl[ \frac{3 \bar{y} \mu (\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{5}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) \\ &\qquad{} + \frac{3\bar{y} (1 - \mu )(\bar{x} - \mu )}{(\bar{r}_{1})^{5}} + \frac{3 \bar{y} \mu (\bar{x} + 1 - \mu )A}{(\bar{r}_{2})^{7}} - \lambda k \\ &\qquad{} - \frac{3}{2}\alpha \bigl( \bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 7}{4}}k + \frac{21}{4}\bar{x}^{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}k \biggr] \\ &\qquad{} + \eta_{0}e^{\lambda t} \biggl[ n^{2} - \frac{(1 - \mu )}{(\bar{r}_{1})^{3}} - \frac{\mu}{(\bar{r}_{2})^{3}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) \\ &\qquad{} + \bar{y}^{2} \biggl\{ \frac{3 (1 - \mu )}{(\bar{r}_{1})^{5}} + \frac{3 \mu}{ (\bar{r}_{2})^{5}} \biggl( 1 + \frac{3A}{2\bar{r}_{{2}}^{2}} \biggr) + \frac{3 \mu A}{\bar{r}_{{2}}^{7}} \biggr\} - \lambda k \\ &\qquad{} + \frac{21\bar{x} \bar{y} \alpha}{4}\bigl(\bar{x}^{2} + \bar{y}^{2} \bigr)^{\frac{ - 11}{4}}k \biggr]. \end{aligned}$$
(13)

Now, from Eqs. (12) and (13), the following simultaneous linear equations can be derived

$$\begin{aligned} &\xi \biggl\{ \lambda^{2} + \frac{1 - \mu}{ (\bar{r}_{1})^{3}} \biggl( 1 - \frac{3 (\bar{x} - \mu )^{2}}{(\bar{r}_{1})^{2}} \biggr) \\ &\quad{}+ \frac{\mu}{(\bar{r}_{2})^{3}} \biggl( 1 - \frac{3 (\bar{x} + 1 - \mu )^{2}}{(\bar{r}_{2})^{2}} \biggr) \\ &\quad{} + \frac{3\mu}{\bar{r}_{2}^{5}}A \biggl( \frac{1}{2} - \frac{3 (\bar{x} + 1 - \mu )^{2}}{2 (\bar{r}_{2})^{2}} - \frac{(\bar{x} + 1 - \mu )^{2}}{(\bar{r}_{2})^{2}} \biggr) - n^{2} \\ &\quad{} - \lambda k + \frac{21\bar{x} \bar{y} \alpha}{4}\bigl(\bar{x}^{2} + \bar{y}^{2} \bigr)^{\frac{ - 11}{4}}k \biggr\} \\ &\quad{} + \eta \biggl\{ - 2\lambda - \frac{3 \bar{y}\mu (\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{5}} - \frac{3\bar{y} (1 - \mu )(\bar{x} - \mu )}{(\bar{r}_{1})^{5}} \\ &\quad{} - \frac{3\mu \bar{y}}{\bar{r}_{2}^{3}}A \biggl( \frac{3 (\bar{x} + 1 - \mu )}{2 (\bar{r}_{2})^{4}} + \frac{(\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{4}} \biggr) - \lambda k \\ &\quad{} - \frac{3}{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2} \bigr)^{\frac{ - 7}{4}}k - \frac{21}{4}\bar{y}^{2}\alpha \bigl( \bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}k \biggr\} = 0 \end{aligned}$$
(14)

and

$$\begin{aligned} &\xi \biggl\{ 2\lambda - \frac{3 \bar{y}\mu (\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{5}} - \frac{3\bar{y} (1 - \mu )(\bar{x} - \mu )}{(\bar{r}_{1})^{5}} \\ &\quad{} - \frac{3\mu \bar{y}}{\bar{r}_{2}^{3}}A \biggl( \frac{3 (\bar{x} + 1 - \mu )}{2 (\bar{r}_{2})^{4}} + \frac{(\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{4}} \biggr) + \lambda k \\ &\quad{} + \frac{3}{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2} \bigr)^{\frac{ - 7}{4}}k - \frac{21}{4}\bar{x}^{2}\alpha \bigl( \bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}k \biggr\} \\ &\quad{} + \eta \biggl\{ \lambda^{2} + \frac{1 - \mu}{ (\bar{r}_{1})^{3}} \biggl( 1 - \frac{3 \bar{y}^{2}}{(\bar{r}_{1})^{2}} \biggr) + \frac{\mu}{(\bar{r}_{2})^{3}} \biggl( 1 - \frac{3 \bar{y}^{2}}{(\bar{r}_{2})^{2}} \biggr) \\ &\quad{} + \frac{3\mu}{\bar{r}_{2}^{5}}A \biggl( \frac{1}{2} - \frac{3 \bar{y}^{2}}{2 (\bar{r}_{2})^{2}} - \frac{\bar{y}^{2}}{(\bar{r}_{2})^{2}} \biggr) - n^{2} \\ &\quad{} - \lambda k - \frac{21\bar{x} \bar{y} \alpha}{4}\bigl(\bar{x}^{2} + \bar{y}^{2} \bigr)^{\frac{ - 11}{4}}k \biggr\} = 0. \end{aligned}$$
(15)

The linear equations (14) and (15) can be written as

$$\begin{aligned} &\xi \bigl( \lambda^{2} + e - h + i - n^{2} - \lambda k_{\bar{x},\dot{\bar{x}}} - k_{\bar{x},\bar{x}} \bigr) \\ &\quad{}+ \eta ( - 2 \lambda - g + j - \lambda k_{\bar{x},\dot{\bar{y}}} - k_{\bar{x},\bar{y}} ) = 0 \end{aligned}$$
(16)
$$\begin{aligned} &\xi ( 2 \lambda - g + j - \lambda k_{\bar{y},\dot{\bar{x}}} + k_{\bar{y},\bar{x}} ) \\ &\quad{} + \eta \bigl( \lambda^{2} + e - f + l - n^{2} - \lambda k_{\bar{y},\dot{\bar{y}}} - k_{\bar{y},\bar{y}} \bigr) = 0 \end{aligned}$$
(17)

where

$$\begin{aligned} &e = \frac{1 - \mu}{(\bar{r}_{1})^{3}} + \frac{\mu}{ (\bar{r}_{2})^{3}}, \end{aligned}$$
(18)
$$\begin{aligned} &f = 3 \biggl[ \frac{1 - \mu}{(\bar{r}_{1})^{5}} + \frac{\mu}{ (\bar{r}_{2})^{5}} \biggr] \bar{y}^{2}, \end{aligned}$$
(19)
$$\begin{aligned} &g = 3 \biggl[ \frac{(1 - \mu )(\bar{x} - \mu )}{(\bar{r}_{1})^{5}} + \frac{\mu (\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{5}} \biggr] \bar{y}, \end{aligned}$$
(20)
$$\begin{aligned} &h = 3 \biggl[ \frac{(1 - \mu )(\bar{x} - \mu )^{2}}{(\bar{r}_{1})^{5}} + \frac{\mu (\bar{x} + 1 - \mu )^{2}}{(\bar{r}_{2})^{5}} \biggr], \end{aligned}$$
(21)
$$\begin{aligned} &i = 3 A \biggl[ \frac{\mu}{2(\bar{r}_{2})^{5}} - \frac{3\mu (\bar{x} + 1 - \mu )^{2}}{2(\bar{r}_{2})^{7}} - \frac{(\bar{x} + 1 - \mu )^{2}}{(\bar{r}_{2})^{7}} \biggr], \end{aligned}$$
(22)
$$\begin{aligned} &j = 3 A \biggl[ \frac{3\mu (\bar{x} + 1 - \mu )}{2(\bar{r}_{2})^{7}} + \frac{\mu (\bar{x} + 1 - \mu )}{(\bar{r}_{2})^{7}} \biggr] \bar{y}, \end{aligned}$$
(23)
$$\begin{aligned} &l = 3 A \biggl[ \frac{\mu}{2(\bar{r}_{2})^{5}} - \frac{3\mu \bar{y}^{2}}{2(\bar{r}_{2})^{7}} - \frac{\mu \bar{y}^{2}}{(\bar{r}_{2})^{7}} \biggr]. \end{aligned}$$
(24)

and

$$\begin{aligned} \begin{aligned}[c] &k_{\bar{x},\bar{x}} = \biggl( \frac{\partial S_{x}}{\partial x} \biggr)_{ -} = \frac{21}{4}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2} \bigr)^{\frac{ - 11}{4}}\bar{x} \bar{y} k,\\ & k_{\bar{x},\dot{\bar{x}}} = \biggl( \frac{S_{x}}{\partial \dot{x}} \biggr)_{ -} = k,\\ & k_{\bar{x},\bar{y}} = \biggl( \frac{\partial S_{x}}{\partial y} \biggr)_{ -} = - k + \frac{21}{4} \bar{y}^{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2} \bigr)^{\frac{ - 11}{4}}k, \\ & k_{\bar{x},\dot{\bar{y}}} = \biggl( \frac{\partial S_{x}}{\partial \dot{y}} \biggr)_{ -} = 0, \\ & k_{\bar{y},\bar{x}} = \biggl( \frac{\partial S_{y}}{\partial x} \biggr)_{ -} = k + \frac{21}{4}\bar{x}^{2}\alpha \bigl(\bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}k,\\ & k_{\bar{y},\dot{\bar{x}}} = \biggl( \frac{\partial S_{y}}{\partial \dot{x}} \biggr)_{ -} = 0,\\ & k_{\bar{y},\bar{y}} = \biggl( \frac{\partial S_{y}}{\partial y} \biggr)_{ -} = \frac{21}{4}\alpha \bigl( \bar{x}^{2} + \bar{y}^{2}\bigr)^{\frac{ - 11}{4}}\bar{x} \bar{y} k, \\ & k_{\bar{y},\dot{\bar{y}}} = \biggl( \frac{\partial S_{y}}{\partial \dot{y}} \biggr)_{ -} = k. \end{aligned} \end{aligned}$$
(25)

Neglecting terms of \(O(k^{2})\), the condition for the determinant of the linear equations defined by Eqs. (16) and (17) to be zero is

$$\begin{aligned} &\lambda^{4} - (k_{{\bar{x},\dot{\bar{x}}}} + k_{\bar{y},\dot{\bar{y}}}) \lambda^{3} + \bigl[2 \bigl(e - n^{2}\bigr) - f - h - k_{\bar{x},\bar{x}} \\ &\quad {}+ 2 (k_{\bar{x},\dot{\bar{y}}} - k_{\bar{y},\dot{\bar{x}}}) - k_{{\bar{y},\bar{y}}} + l + i + 4 - (k_{\bar{x},\dot{\bar{y}}}k_{\bar{y},\dot{\bar{x}}} \\ &\quad {}+ k_{\bar{x},\dot{\bar{x}}}k_{\bar{y},\dot{\bar{y}}})\bigr] \lambda^{2} + \bigl[\bigl(n^{2} - e + f\bigr)k_{{\bar{x},\dot{\bar{x}}}} + (i - e + h)k_{\bar{y},\dot{\bar{y}}} \\ &\quad {}+ 2 (k_{{\bar{x},\bar{y}}} - k_{\bar{y},\bar{x}}) + n^{2}k_{\bar{y},\dot{\bar{y}}} - g (k_{\bar{x},\dot{\bar{y}}} + k_{\bar{y},\dot{\bar{x}}}) \\ &\quad {}+ j (k_{\bar{x},\dot{\bar{y}}} + k_{\bar{y},\dot{\bar{x}}}) - l k_{\bar{x},\dot{\bar{x}}} + (k_{{\bar{x},\dot{\bar{x}}}}k_{\bar{y},\bar{y}} + k_{\bar{x},\bar{x}}k_{\bar{y},\dot{\bar{y}}}) \\ &\quad {} - (k_{\bar{x},\dot{\bar{y}}}k_{\bar{y},\bar{x}} + k_{\bar{y},\dot{\bar{x}}}k_{\bar{x},\bar{y}})\bigr] \lambda + \bigl[\bigl(e - h - n^{2}\bigr) \bigl(e - f - n^{2} \bigr) \\ &\quad {} - g^{2} + \bigl(n^{2} - e + f\bigr)k_{\bar{x},\bar{x}} + \bigl(n^{2} - e + h\bigr)k_{\bar{y},\bar{y}} \\ &\quad {} - g (k_{\bar{x},\bar{y}} + k_{\bar{y},\bar{x}}) + l\bigl(e - h + i - n^{2}\bigr) + i \bigl(e - f - n^{2}\bigr) \\ &\quad {} + j (k_{\bar{x},\bar{y}} + k_{\bar{y},\bar{x}}) - l k_{\bar{x},\bar{x}} - i k_{\bar{y},\bar{y}} + (k_{\bar{x},\bar{x}}k_{\bar{y},\bar{y}} - k_{\bar{x},\bar{y}}k_{\bar{y},\bar{x}}) \\ &\quad {} + 2 g j - j^{2}\bigr] = 0 \end{aligned}$$
(26)

This quadratic equation (26) has the general form

$$ \lambda^{4} + \sigma_{3} \lambda^{3} + ( \sigma_{20} + \sigma_{2} ) \lambda^{2} + \sigma_{1}\lambda + ( \sigma_{00} + \sigma_{0} ) = 0 $$
(27)

where

$$\begin{aligned} \sigma_{0} &= \bigl(n^{2} - e + f\bigr)k_{\bar{x},\bar{x}} + \bigl(n^{2} - e + h\bigr)k_{\bar{y},\bar{y}} - g (k_{\bar{x},\bar{y}} + k_{\bar{y},\bar{x}}) \\ &\quad {} + l\bigl(e - h + i - n^{2}\bigr) + i \bigl(e - f - n^{2}\bigr) + j (k_{\bar{x},\bar{y}} + k_{\bar{y},\bar{x}}) \\ &\quad {} - l k_{\bar{x},\bar{x}} - i k_{\bar{y},\bar{y}} + (k_{\bar{x},\bar{x}}k_{\bar{y},\bar{y}} - k_{\bar{x},\bar{y}}k_{\bar{y},\bar{x}}) + 2 g j - j^{2}),\\ \sigma_{1} &= \bigl(n^{2} - e + f\bigr)k_{\bar{x},\dot{\bar{x}}} + (i - e + h)k_{\bar{y},\dot{\bar{y}}} + 2 (k_{\bar{x},\bar{y}} - k_{\bar{y},\bar{x}}) \\ &\quad {} + n^{2}k_{\bar{y},\dot{\bar{y}}} - g (k_{\bar{x},\dot{\bar{y}}} + k_{\bar{y},\dot{\bar{x}}}) + j (k_{\bar{x},\dot{\bar{y}}} + k_{\bar{y},\dot{\bar{x}}}) \\ &\quad {}- l k_{\bar{x},\dot{\bar{x}}} + (k_{\bar{x},\dot{\bar{x}}}k_{\bar{y},\bar{y}} + k_{\bar{x},\bar{x}}k_{\bar{y},\dot{\bar{y}}}) - (k_{\bar{x},\dot{\bar{y}}}k_{\bar{y},\bar{x}} + k_{\bar{y},\dot{\bar{x}}}k_{\bar{x},\bar{y}}), \\ \sigma_{2} &= - k_{\bar{y},\bar{y}} - k_{\bar{x},\bar{x}} + 2 (k_{\bar{x},\dot{\bar{y}}} - k_{\bar{y},\dot{\bar{x}}}) + l + i + 4 - (k_{\bar{x},\dot{\bar{y}}}k_{\bar{y},\dot{\bar{x}}} \\ &\quad {} + k_{\bar{x},\dot{\bar{x}}}k_{\bar{y},\dot{\bar{y}}}), \\ \sigma_{3}& = - k_{\bar{x},\dot{\bar{x}}} - k_{\bar{y},\dot{\bar{y}}}, \end{aligned}$$
$$\begin{aligned} \sigma_{20} &= 2\bigl(e - n^{2}\bigr) - f - h, \\ \sigma_{00} &= \bigl(e - h - n^{2}\bigr) \bigl(e - f - n^{2}\bigr) - g^{2}. \end{aligned}$$

The values of \(\sigma_{00}\), \(\sigma_{20}\) and \(\sigma_{i}\) \((i = 0, 1, 2, 3)\) can be obtained by evaluating \(e,f,g\) and \(h\)defined earlier. The value of the coefficient in the zero drag case is denoted by adding additional subscript 0.

$$\begin{aligned} \begin{aligned}[c] &\sigma_{00} = \frac{27}{4}\mu + \frac{9}{2}A - 9\mu A + \frac{3\sqrt{3}}{4}k, \\ &\sigma_{20} = - 3 - \frac{\sqrt{3}}{A}k + \frac{2}{\sqrt{3}} k, \\ &\sigma_{0} = \frac{27}{16}A + \frac{9}{4}\mu A + \frac{3\sqrt{3}}{8}k, \\ &\sigma_{1} = 2k - \frac{3}{2}A k + \frac{33}{8}\mu A k, \\ &\sigma_{2} = 4 - \frac{3}{4}A - \frac{15}{4}\mu A, \\ &\sigma_{3} = - 2k. \end{aligned} \end{aligned}$$
(28)

By assuming \(\sigma_{i}\) to be small, we investigate the stability of the non zero drag case. We can use the classical solutions of the zero drag case (i.e. when \(k =0\)). Equation (27) reduces to

$$ \lambda^{4} + \sigma_{20}\lambda^{2} + \sigma_{00} = 0. $$
(29)

The four classical solutions for \(L_{4}\) and \(L_{5}\) to \(O(\mu )\)are given by the pair of values

$$ \begin{aligned}[c] L_{4,5}{:}\quad & \lambda_{1,2} = \pm \sqrt{ - 1 + \frac{27}{4} \mu} \\ &\lambda_{3,4} = \pm \sqrt{ - \frac{27}{4}\mu} \end{aligned} $$
(30)

The four roots of the classical characteristic equation can be written as

$$ \lambda_{n} = \pm \mathrm{T} i\quad (n = 1,\ldots,4) $$
(31)

where

$$ \mathrm{T} = \sqrt{\frac{\sigma_{20 \pm} \sqrt{\sigma_{20}^{2} - 4\sigma_{00}}}{2}} $$
(32)

is a real quantity for \(L_{4}\) and \(L_{5}\). Using the values of \(\sigma_{00}\) and \(\sigma_{20}\) given in Eqs. (28), we have

$$ \mathrm{T}^{2} = 1 - \frac{27}{4}\mu\quad \mbox{and}\quad \mathrm{T}^{2} = \frac{27}{4}\mu $$
(33)

In the case of drag, we assume a solution of the form

$$\begin{aligned} \lambda &= \lambda_{n}(1 + \rho + \upsilon i) \\ &= \bigl[ \mp \upsilon \pm (1 + \rho ) i\bigr] \mathrm{T} \end{aligned}$$
(34)

where \(\rho\) and \(\upsilon\) are small real quantities. To lowest order we have

$$\begin{aligned} &\lambda^{2} = \bigl[ - (1 + 2 \rho ) - 2 \upsilon i)\bigr] \mathrm{T}^{2} \end{aligned}$$
(35)
$$\begin{aligned} & \lambda^{3} = \bigl[ \pm 3 \upsilon \mp (1 + 3 \rho ) i)\bigr] \mathrm{T}^{3} \end{aligned}$$
(36)
$$\begin{aligned} &\lambda^{4} = \bigl[(1 + 4 \rho ) + 4 \upsilon i)\bigr] \mathrm{T}^{4} \end{aligned}$$
(37)

Substituting these in equation (27), and neglecting products of \(\rho\) or \(\upsilon\) with \(\sigma_{i}\), and solving the real and imaginary parts of the resulting simultaneous equations for \(\rho\) or \(\upsilon\) we get

$$\begin{aligned} &\upsilon = \frac{ \pm \sigma_{3}\mathrm{T}^{2} \mp \sigma_{1}}{2\mathrm{T} (2\mathrm{T}^{2} - \sigma_{20})}, \end{aligned}$$
(38)
$$\begin{aligned} &\rho = \frac{(\sigma_{00} + \sigma_{0}) - (\sigma_{20} + \sigma_{2}) \mathrm{T}^{2} + \mathrm{T}^{4}}{2\mathrm{T}^{2}(\sigma_{20} - 2\mathrm{T}^{2})}. \end{aligned}$$
(39)

(i) The stability of \(L_{4}\)

For \(L_{4}\), we have

$$\begin{aligned} &\upsilon = \frac{\sigma_{3}\mathrm{T}^{2} - \sigma_{1}}{2\mathrm{T} (2\mathrm{T}^{2} - \sigma_{20})}, \end{aligned}$$
(40)
$$\begin{aligned} &\rho = \frac{(\sigma_{00} + \sigma_{0}) - (\sigma_{20} + \sigma_{2}) \mathrm{T}^{2} + \mathrm{T}^{4}}{2\mathrm{T}^{2}(\sigma_{20} - 2\mathrm{T}^{2})}. \end{aligned}$$
(41)

On putting the values of \(\sigma_{i}\), in Eqs. (40) and (41) from Eq. (28) and also taking, \(\mathrm{T}^{2} = \frac{27}{4}\mu\), we have

$$\begin{aligned} &\upsilon = \frac{k( - 16 - 108\mu )}{36\sqrt{3\mu} (2 + 9\mu )} + \frac{k(12 + 33\mu )A}{36\sqrt{3\mu} (2 + 9\mu )},\\ & \rho = \biggl( \frac{ - 11\sqrt{3} - 162\mu}{72(2 + 9\mu )} + \frac{( - 11 + 3\mu - 45\mu^{2})A}{36\mu (2 + 9\mu )^{2}} \biggr) \\ &\qquad{} +\biggl( \frac{ - 18\sqrt{3} - 33\mu - 9\sqrt{3} \mu + 162\sqrt{3} \mu^{2}}{162(2 + 9\mu )^{2}}k \\ &\qquad {}+ \frac{( - 22\sqrt{3} + 6\sqrt{3} \mu - 90\sqrt{3} \mu^{2})A}{162(2 + 9\mu )^{2}}k \biggr). \end{aligned}$$

Now, putting these values of \(\rho\) and \(\upsilon\) in Eq. (37), and neglecting the terms of \(O(k\mu )\), we get the characteristic equation as

$$\begin{aligned} &\lambda^{4} - \Bigl(27\mu ( - 24\sqrt{3} k + 216\mu - 66\sqrt{3} \mu - 44k \mu - 12\sqrt{3} \mu k \\ &\quad {}+ 972\mu^{2} - 297\sqrt{3} \mu^{2} + 216\sqrt{3} \mu^{2} k\Bigr)\big/\bigl(32(2 + 9\mu )^{2}\bigr)\\ &\quad {} - \Bigl(9\{ \mu (18 + 4\sqrt{3} k + 81\mu )(11 - 3\mu + 45\mu^{2})\} A\Bigr)\\ &\quad {}\big/\bigl(16(2 + 9\mu )^{2}\bigr) = 0 \end{aligned}$$

whose roots are

$$\begin{aligned} &\lambda_{1} = - \biggl[ - \frac{891\mu A}{2(2 + 9\mu )^{2}} + \frac{7533\mu^{2}A}{16(2 + 9\mu )^{2}} + \frac{729\mu^{2}}{4(2 + 9\mu )^{2}} \biggr]^{\frac{1}{4}},\\ & \lambda_{2} = - i \biggl[ - \frac{891\mu A}{2(2 + 9\mu )^{2}} + \frac{7533\mu^{2}A}{16(2 + 9\mu )^{2}} + \frac{729\mu^{2}}{4(2 + 9\mu )^{2}} \biggr]^{\frac{1}{4}}, \\ &\lambda_{3} = i \biggl[ - \frac{891\mu A}{2(2 + 9\mu )^{2}} + \frac{7533\mu^{2}A}{16(2 + 9\mu )^{2}} + \frac{729\mu^{2}}{4(2 + 9\mu )^{2}} \biggr]^{\frac{1}{4}},\\ &\lambda_{4} = \biggl[ - \frac{891\mu A}{2(2 + 9\mu )^{2}} + \frac{7533\mu^{2}A}{16(2 + 9\mu )^{2}} + \frac{729\mu^{2}}{4(2 + 9\mu )^{2}} \biggr]^{\frac{1}{4}}. \end{aligned}$$

Also on taking \(T^{2} = 1 - \frac{27}{4}\mu\) in Eqs. (40) and (41) from Eq. (28), we get the characteristic equation as

$$\begin{aligned} &\lambda^{4} + \Big(( - 4 + 27\mu )(2400 - 440\sqrt{3} - 30348\mu - 6102\sqrt{3} \mu \\ &\quad {}+ 61236\mu^{2} + 13851\sqrt{3} \mu^{2})\Big)\big/\big(96( - 10 + 27\mu )^{2}\big)\\ &\quad {} + \Big(( - 4 + 27\mu )(1110 - 148\sqrt{3} k - 4287\mu - 172\sqrt{3} \mu k\\ &\quad {} - 567\mu^{2} - 540\sqrt{3} \mu^{2}k + 10935\mu^{3})\Big)\\ &\quad {}\big/\big(16( - 10 + 27\mu )^{2}\big) + \frac{8}{5}ik - \frac{3}{5}A i k = 0. \end{aligned}$$

whose roots are

$$\begin{aligned} \lambda_{1} &= - \biggl[ - \frac{(8 - 3A) i k}{5} + \biggl( \frac{100}{(10 - 27\mu )^{2}} - \frac{55}{\sqrt{3} (10 - 27\mu )^{2}}\\ &\quad {} + \frac{555A}{2(10 - 27\mu )^{2}} - \frac{3879\mu}{2(10 - 27\mu )^{2}} - \frac{261\sqrt{3} \mu}{2(10 - 27\mu )^{2}} \\ &\quad {}- \frac{23559A\mu}{8(10 - 27\mu )^{2}} \biggr) \biggr]^{\frac{1}{4}},\\ \lambda_{2} &= - \biggl[ \frac{(8 - 3A) k}{5} + \biggl( \frac{100}{(10 - 27\mu )^{2}} - \frac{55}{\sqrt{3} (10 - 27\mu )^{2}}\\ &\quad {}+ \frac{555A}{2(10 - 27\mu )^{2}} - \frac{3879\mu}{2(10 - 27\mu )^{2}} - \frac{261\sqrt{3} \mu}{2(10 - 27\mu )^{2}} \\ &\quad {}- \frac{23559A\mu}{8(10 - 27\mu )^{2}} \biggr) i \biggr]^{\frac{1}{4}}, \\ \lambda_{3} &= \biggl[ \frac{(8 - 3A) k}{5} + \biggl( \frac{100}{(10 - 27\mu )^{2}} - \frac{55}{\sqrt{3} (10 - 27\mu )^{2}} \\ &\quad {}+ \frac{555A}{2(10 - 27\mu )^{2}} - \frac{3879\mu}{2(10 - 27\mu )^{2}} - \frac{261\sqrt{3} \mu}{2(10 - 27\mu )^{2}}\\ &\quad {} - \frac{23559A\mu}{8(10 - 27\mu )^{2}} \biggr) i \biggr]^{\frac{1}{4}}, \\ \lambda_{4} &= \biggl[ - \frac{(8 - 3A) i k}{5} + \biggl( \frac{100}{(10 - 27\mu )^{2}} - \frac{55}{\sqrt{3} (10 - 27\mu )^{2}}\\ &\quad{} + \frac{555A}{2(10 - 27\mu )^{2}} - \frac{3879\mu}{2(10 - 27\mu )^{2}} - \frac{261\sqrt{3} \mu}{2(10 - 27\mu )^{2}} \\ &\quad{}- \frac{23559A\mu}{8(10 - 27\mu )^{2}} \biggr) \biggr]^{\frac{1}{4}}. \end{aligned}$$

If \(\upsilon \ne 0\),

According to Murray (1994), the resulting motion of a particle is asymptotically stable only when all the real parts of \(\lambda\)are negative and the condition for asymptotically stable under the arbitrary drag force is given by

$$ 0 < \sigma_{1} < \sigma_{3} $$
(42)

where \(\sigma_{1}\) and \(\sigma_{3}\) are defined in Eq. (26). But we see that the linear stability of triangular equilibrium points does not depend on the value of \(k_{x,x} \) and \(k_{y,y}\). Therefore the condition \(\sigma_{3} > 0\) can only be satisfied when \(k\) is positive and the drag force is a function of \(\dot{x}\) and \(\dot{y}\).

But here in our case of Stokes drag \(\sigma_{1} = 2k\), \(\sigma_{3} = - 2k\) and therefore \(\sigma_{1} > \sigma_{3}\) and hence \(L_{4}\) is not asymptotically stable. Further one of the roots of \(\lambda\) i.e. \(\lambda_{4}\) has positive real root. Therefore \(L_{4}\) is not stable. Thus we conclude that \(L_{4}\) is neither stable nor asymptotically stable and hence linearly unstable.

Similarly, we conclude that \(L_{5}\) is neither stable nor asymptotically stable and hence linearly unstable.

5 Conclusion

In the present paper, we have considered the smaller primary as an oblate spheroid and bigger one as a point mass. It is observed that there exist two non-collinear libration points \(L_{4,5}(\bar{x},\bar{y})\) (Eq. (9)).

Under the effect of Stokes drag, we have derived a set of linear equations (Eqs. (16) and (17)), from which we derive a characteristic equation having the general form (Eq. (27)). Thereafter, we have derived the approximate expressions for \(\sigma_{0}\), \(\sigma_{1}\), \(\sigma_{2}\), \(\sigma_{3}\), \(\sigma_{00}\) and \(\sigma_{20}\) occurring in the above characteristic equation. These expressions are given in terms of the partial derivatives of the Stokes drag, evaluated at the libration points.

In the case of drag force, by using the terminology of Murray, we assume a solution of the form (Eq. (34)). Where \(\upsilon\) and \(\rho\) are small real quantities and

$$\lambda_{n} = \pm \mathrm{T} i\quad (n = 1,\ldots,4) $$

is a real quantity for \(L_{4}\) and \(L_{5}\) in the classical case. The values of \(\upsilon\) and \(\rho\) (Eqs. (38), (39)) have been obtained by substituting the values of \(\lambda\), \(\lambda^{2}\), \(\lambda^{3}\) and \(\lambda^{4}\) in the characteristic equation.

By using Murray terminology, to investigate the stability of the shifted points, the resulting motion of a particle is asymptotically stable only when all the real parts of \(\lambda\) are negative and the condition for asymptotical stability under the drag force is given by (Eq. (42)).

The condition \(\sigma_{3} > 0\) can only be satisfied when \(k > 0\). In the case of Stokes drag \(\sigma_{1} = 2k\) and \(\sigma_{3} = - 2k\) therefore Eq. (42) is not satisfied. Therefore \(L_{4}\) and \(L_{5}\) are not asymptotically stable. Further we have seen that one of the roots of \(\lambda\) i.e. \(\lambda_{4}\) has positive real root, thus \(L_{4}\) and \(L_{5}\) are not stable. Hence due to Stokes drag, \(L_{4}\) and \(L_{5}\) are neither stable nor asymptotically stable but unstable whereas in the classical case \(L_{4}\) and \(L_{5}\) are stable for the mass ratio \(\mu < 0.03852\) (Szebehely 1967).

In the case of Stokes drag effect (both the primaries are point masses), when \(k = 0\) the results obtained are in conformity with the classical problem (Szebehely 1967). When \(A = 0\) (smaller primary is an oblate spheroid and bigger one is a point mass), the results obtained are in conformity with those of Jain and Aggarwal (2015).