1 Introduction

Echoing soliton concepts that flourished in the multi-disciplinary since a few decades ago, rogue wave has attracted recently the attention of researchers in various physical settings, e.g., in hydrodynamics [1, 2], capillary waves [3, 4], plasma physics [5], nonlinear optics [6] and Bose–Einstein condensation [7]. Due to their harm to various hydrotechnic constructions, the investigation of rogue waves becomes a very important problem [8, 9]. Rogue waves, which have a peak amplitude generally more than twice the significant wave height, are thought to appear from nowhere and disappear without a trace in the ocean and most experimental optical systems [10–13]. Among various models describing such waves, the focusing nonlinear Schrödinger (NLS) equation [14, 15] is the most accepted one. The NLS equation admits a type of rational solution that is localized in both space and time, i.e., the Peregrine soliton [16]. This simplest rational solution of the NLS equation, which was first predicted as far as 30 years ago [16], is frequently used as a model of a rogue wave [9, 17]. Breather solutions are presently regarded as potential prototypes for the rogue waves in many fields of physics [2, 3, 18–20]. Breathers develop due to the instability of small-amplitude perturbations that may grow in size to disastrous proportions [21]. There are two types of breathers, namely the Kuznetsov–Ma breathers (KMBs) [22] and Akhmediev breathers (ABs) [23]. The KMBs are periodic in space and localized in time, while the ABs are periodic in time and localized in space [22, 23]. Taking the period of both solutions to infinity leads to a first-order doubly localized Peregrine soliton.

The modulational instability (MI) is generally considered to be one of factors which may give rise to rogue-wave excitation [16]. As a fundamental characteristic of many nonlinear dispersive systems, MI is related to the growth of periodic perturbations on an unstable continuous-wave background [24]. A rogue wave may be the result of MI, but not every type of MI results in rogue-wave generation [25–27]. One of theoretical researches has revealed the close relationship between rogue waves and baseband MI, i.e., MI whose bandwidth includes components of arbitrarily low frequency [28–30]. Another possible explanation presented by Zhao and Ling is that rogue wave comes from MI under the “resonance” perturbation with continuous-wave background [31].

Recent studies have revealed the intricate link between the rogue waves (or breathers) and solitons of a certain class of nonlinear evolution equations. When the eigenvalues meet a certain locus on the complex plane, Akhmediev et al. have discovered that the breather solutions of the Hirota [32] equation and fifth-order NLS [33] equation can be converted into soliton solutions on a background, which does not exist in the standard NLS equation. He et al. have reported that the rational solution of a mixed NLS equation can describe five soliton states, including a paired bright–bright soliton, a single soliton, a paired bright-gray soliton, a paired bright-black soliton and a rogue-wave state [34]. They have found that the state transition among these five states is induced by tuning the effects of self-steepening and self-phase modulation. Liu et al. have shown that the breathers can be converted into different types of nonlinear waves in the coupled NLS-MB system, including the multi-peak soliton, periodic wave, antidark soliton and W-shaped soliton [35]. In particular, they have indicated that the transition between the rogue waves and W-shaped solitons of the Hirota and coupled Hirota equations occurs as a result of the attenuation of MI growth rate to vanish in the zero-frequency perturbation region [36, 37].

In this paper, we study the AB system [38, 39],

$$\begin{aligned}&\left( \frac{\partial }{\partial T}+c_{1} \frac{\partial }{\partial X}\right) \left( \frac{\partial }{\partial T}+c_{2}\frac{\partial }{\partial X}\right) A= l_{1}A-l_{2}A B,\nonumber \\&\left( \frac{\partial }{\partial T}+c_{2}\frac{\partial }{\partial X}\right) B=\left( \frac{\partial }{\partial T}+c_{1}\frac{\partial }{\partial X}\right) |A|^{2}, \end{aligned}$$
(1.1)

where A is the amplitude of the wave packet and B is a quantity measuring the correction of the basic flow due to the wave packet, T and X denote the time and space variables, respectively, \(c_{1}\) and \(c_{2}\) stand for two group velocities of the underlying linear problem, and \(l_{1}\) represents a parameter measuring the state of the basic flow. When the basic flow is supercritical, \( l_{1} > 0\), and when the basic flow is subcritical, \(l_{1} < 0\). The parameter \(l_{2}\) reflects the interaction of the wave packet and the meanflow, and it is always positive. System (1.1) describes marginally unstable baroclinic wave packets in geophysical fluids and also ultrashort pulses in nonlinear optics.

As shown in Ref. [40], System (1.1) can be reduced to a simpler form

$$\begin{aligned} A_{xt}=\alpha A+\beta A B,\quad B_{x}=-\frac{1}{2}\gamma (|A|^{2})_{t}, \end{aligned}$$
(1.2)

with

$$\begin{aligned} \alpha =\frac{l_{1} c_{2}}{(c_{1}-c_{2})^{2}},\quad \beta =-\frac{l_{2} c_{2}}{(c_{1}-c_{2})^{2}},\quad \gamma =-\frac{2}{c_{2}},\nonumber \\ \end{aligned}$$
(1.3)

by the transformations

$$\begin{aligned} x=X-c_{1}T ,\quad t=T-\frac{X}{c_{2}}. \end{aligned}$$

Recently, certain properties of System (1.2) have been investigated. System (1.2) can be transformed to the Sine-Gordon equation when A is the real value and to the self-induced transparency system when A is the complex value [38, 39]. Lax pair and some periodic solutions of System (1.2) have been derived in Ref. [39]. MI and breather dynamics of System (1.2) have been discussed in Ref. [41]. Reference [42] has studied the envelope solitary waves and periodic waves of System (1.2). The higher-order rogue-wave solutions of System (1.2) have been found via the modified Darboux transformation (mDT) in Ref. [43]. The semirational solutions have been derived in Ref. [30], and the link between the baseband MI and the existence condition of these rogue waves has been revealed. Recently, the rogue waves and MI have been demonstrated for a coupled AB system, i.e., a wave–current interaction model describing baroclinic instability processes in geophysical flows [40].

The aim of the present paper is to study the breather–soliton dynamics of System (1.2). We present intriguing different kinds of nonlinear localized and periodic waves, including the multi-peak soliton, M-shaped soliton, W-shaped soliton and periodic wave. Interestingly, these waves are stationary nonlinear waves with respect to x-axis. Further, due to the nonpropagating characteristic, the nonlinear interactions show some novel properties. The breather-to-soliton transition is related to a special type of MI analysis that involves a MI stability region in a low perturbation frequency region.

The arrangement of the paper is as follows: In Sect. 2, we present different types of stationary nonlinear waves of System (1.2), including the multi-peak soliton, M-shaped soliton, W-shaped soliton and periodic waves. And the transition condition will be analytically given. The properties of interactions between different types of nonlinear waves are graphically studied in Sect. 3. The connection between the MI growth rate and transition condition is revealed in Sect. 4. Finally, Sect. 5 gives the conclusions of this paper.

2 Different types of stationary nonlinear waves

In this section, we mainly study the transitions between the first-order breather and various nonlinear waves for System (1.2). The first-order breather solution of System (1.2) is given [30]

$$\begin{aligned}&A_{B}^{[1]}=\left( a+n_{1}\frac{G_{B}^{[1]}+i H_{B}^{[1]}}{D_{B}^{[1]}}\right) e^{i \rho } ,\nonumber \\&B_{B}^{[1]}=b+\frac{4 i}{\beta } \left( \frac{m_{1}E_{B}^{[1]}+i n_{1} F_{B}^{[1]}}{D_{B}^{[1]}}\right) _{t}, \end{aligned}$$
(2.1)

with

$$\begin{aligned}&\rho =\omega x+k t,\qquad k=-\frac{\alpha +b \beta }{\omega }, \\&G_{B}^{[1]}=k_{1}k_{2}\cos (t V_{H}+x h_{R})\cosh (2 \chi _{I})\\&\quad \qquad \cosh (t V_{T}+x h_{I})\sin (2 \chi _{R}),\nonumber \\&H_{B}^{[1]}=k_{1} k_{2} \sin (t V_{H}+x h_{R})\sinh (2 \chi _{I})\\&\quad \qquad +\cos (2 \chi _{R})\sinh (t V_{T}+x h_{I}),\nonumber \\&D_{B}^{[1]}=-k_{1} k_{2} \cos (t V_{H}+x h_{R})\sin (2 \chi _{R})\\&\quad \qquad +\cosh (t V_{T}+x h_{I})\cosh (2 \chi _{I}) ,\nonumber \\&E_{B}^{[1]}=k_{1} k_{2}\cos (t V_{H}+x h_{R})\sin (2 \chi _{R})\\&\qquad \quad -\cosh (t V_{T}+x h_{I})\cosh (2 \chi _{I}),\nonumber \\&F_{B}^{[1]}=k_{1} k_{2} \sin (t V_{H}+x h_{R})\cos (2 \chi _{R})\\&\quad \qquad -\sinh (t V_{T}+x h_{I})\sinh (2 \chi _{I}),\nonumber \\&h=\sqrt{a^{2}\beta \gamma +(2 \lambda +\omega )^{2}}=h_{R}+ih_{I}, \\&\chi =\frac{1}{2}\arccos \left( \frac{h}{2}\right) ,k_{1}=1,k_{2}=\pm 1,\lambda =m+in,\\&\varpi =\frac{h}{2}\left( x+\frac{k}{2\lambda }t\right) =(x+(\varpi _R+i \varpi _I)t)\frac{h}{2},\nonumber \nonumber \\&V_{T}=2(\varpi _{R}h_{I}+\varpi _{I}h_{R}),\nonumber \\&V_{H}=2(\varpi _{R}h_{R}-\varpi _{I}h_{I}).\nonumber \end{aligned}$$

From the above expression, one can find that the breather solution (2.1) comprises the hyperbolic functions \(\sinh (t V_{T}+x h_{I})\), \(\cosh (t V_{T}+x h_{I})\) and the trigonometric functions \(\sin (t V_{H}+x h_{R})\), \(\cos (t V_{H}+x h_{R})\), where \(\varpi _{R}+\frac{\varpi _{I}h_{R}}{h_{I}}\) and \(\varpi _{R}-\frac{\varpi _{I}h_{I}}{h_{R}}\) are the corresponding velocities. The hyperbolic functions and trigonometric functions, respectively, characterize the localization and the periodicity of the transverse distribution t of those waves. The nonlinear wave described by solution (2.1) can be deemed to a nonlinear combination of a soliton and a periodic wave with the velocities \(\varpi _{R}+\frac{\varpi _{I}h_{R}}{h_{I}}\) and \(\varpi _{R}-\frac{\varpi _{I}h_{I}}{h_{R}}\). In the following, we show that solution (2.1) can describe different kinds of nonlinear wave states depending on the values of velocity difference \(\frac{\varpi _{I}(h_{R}^{2}+h_{I}^{2})}{h_{R}h_{I}}\).

When \(\varpi _{I}(\frac{h_{R}^{2}+h_{I}^{2}}{h_{R}h_{I}})\ne 0\) (or \(\varpi _{I}\ne 0\)), solution (2.1) characterizes the localized waves with breathing behavior on constant backgrounds (i.e., the breathers and rogue waves). Further, if \(m=-\frac{\omega }{2}\), we have the ABs with \(|n|<|\frac{a\sqrt{\beta \gamma }}{2}|\), the KMBs with \(|n|>|\frac{a\sqrt{\beta \gamma }}{2}|\) and the Peregrine soliton with \(|n|=|\frac{a\sqrt{\beta \gamma }}{2}|\). Those solutions have been obtained in Refs. [30, 41, 43].

Fig. 1
figure 1

A breather transformed into a multi-peak soliton with \(a=2, b=1, \gamma =1, \omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=-1\) and \(\lambda _{1}=\lambda _{2}^{*}=7.2-0.7 i\). b Is the contour plot of a. c Is the cross-sectional view of a at \(t=0\)

Full size image
Fig. 2
figure 2

A breather transformed into an oscillation W-shaped soliton with \(a=2, b=1, \gamma =1,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=-1\) and \(\lambda _{1}=\lambda _{2}^{*}=-0.8-0.7 i\). b Is the contour plot of a. c Is the cross-sectional view of a at \(t=0\)

Full size image

Conversely, if \(\varpi _{I}=0\), the soliton and a periodic wave in solution (2.1) have the same velocity \(\varpi _{R}\). Meanwhile, we should point out that the case \(\varpi _{I}=0\) is equivalent to the following condition

$$\begin{aligned} \frac{V_{T}}{h_{I}}=\frac{V_{H}}{h_{R}} , \end{aligned}$$
(2.2)

i.e.,

$$\begin{aligned} b=-\frac{\alpha }{\beta } . \end{aligned}$$
(2.3)

Equation (2.2) means the extrema of trigonometric and hyperbolic functions in solution (2.1) are located along the same straight lines in the (x, t) plane, which results in the transformation of the breather into a continuous soliton. Additionally, the case \(b=-\frac{\alpha }{\beta }\) is also equivalent to \(k=0\). Then, the parameters \(\varpi _R\), \(\varpi _I\), \(V_T\) and \(V_H\) vanish, and solution (2.1) only depends on x and is independent of t. Thus, the nonlinear waves described by solution (2.1) possess the nonpropagating characteristic.

Under the transition condition (2.3), we demonstrate several kinds of transformed nonlinear waves on constant background for A, including the multi-peak solitons, M-shaped soliton, periodic waves and W-shaped soliton. We omit the results in the B component, since it only describes the plane wave in the case of \(k=0\). Figure 1 shows a multi-peak soliton on constant background that does not propagate along x direction. Such structure is composed of a soliton and periodic wave. Adding the absolute value of real part to the eigenvalue, |m|, we find that the peak numbers of multi-peak localized structure decrease, as depicted in Fig. 2. From the cross-sectional views, it is observed that both of those multi-peak solitons have one main peak which is located at (0, 0). The maximum amplitude of \(|A|^{2}\) at (0, 0) can be given analytically

$$\begin{aligned} |A(0,0)|^{2}=\left( a-4 \frac{n}{\sqrt{\beta \gamma }}\right) ^{2} . \end{aligned}$$
(2.4)

Equation (2.4) indicates that the imaginary part of eigenvalue (n) has real effects on the maximum amplitude of \(|A|^{2}\) at (0, 0).

Fig. 3
figure 3

A breather transformed into a M-shaped soliton with \(a=2, b=1, \gamma =1,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=-1\) and \(\lambda _{1}=\lambda _{2}^{*}=-0.8+i\). b Is the contour plot of a. c Is the cross-sectional view of a at \(t=0\)

Full size image

Further, the structure in Fig. 2 (we call it an oscillation W-shaped soliton) corresponds to the case \(|A(0, 0)|^{2}_{xx}>0\) with the appropriate value of n (\(n=-0.7\)). In other words, the coordinate origin (0, 0) is a local maximum of \(|A(x, 0)|^{2}\). Nevertheless, when the value of n exceeds a certain range, \(|A(0, 0)|^{2}_{xx}\) is less than zero, which means that the coordinate origin (0, 0) is a local minimum of \(|A(x, 0)|^{2}\). In this case, the oscillation W-shaped soliton translates into a M-shaped soliton shown in Fig. 3. It is observed that this structure has two main peaks with identical amplitudes. In order to reveal the effect of the value of n on \(|A(0, 0)|^{2}_{xx}\), we plot Fig. 4 with \(a=2, b=1, \gamma =1,\omega =1, \alpha =-1,\beta =1,k_{1}=1,k_{2}=-1\) and \( m=-0.8\). If \(0<n<1.25\), namely, \(|A(0, 0)|^{2}_{xx}<0\), the solution (2.1) describes the M-shaped soliton, whereas it describes the W-shaped one.

Fig. 4
figure 4

Effects of the imaginary part of eigenvalue n on \(|A(0, 0)|^{2}_{xx}\) with \(a=2, b=1, \gamma =1,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=-1\) and \(m=-0.8\). Two zeros of the \(|A(0, 0)|^{2}_{xx}\) are in (0, 0) and (1.25, 0), respectively

Full size image
Fig. 5
figure 5

A breather transformed into an antidark soliton with \(a=2, b=1,\gamma =1,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=1\) and \(\lambda _{1}=\lambda _{2}^{*}=-0.5-1.8 i\). b Is the contour plot of a. c Is the cross-sectional view of a at \(t=0\)

Full size image
Fig. 6
figure 6

A breather transformed into a periodic wave with \(a=2, b=1, \gamma =1,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=-1\) and \(\lambda _{1}=\lambda _{2}^{*}=-0.5+0.7 i\). b Is the contour plot of a. c Is the cross-sectional view of a at \(t=0\)

Full size image
Fig. 7
figure 7

A rogue wave transformed into a W-shaped soliton with \(a=2, b=1, \gamma =1,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=1\) and \( \lambda _{1}=\lambda _{2}^{*}=-0.5+i.\) b Is the contour plot of a. c Is the cross-sectional view of a at \(t=0\)

Full size image

Next, we derive two special nonlinear waves from the solution (2.1), i.e., the antidark soliton and periodic wave. The former exists in isolation when \(h_{R}\) vanishes, while the latter independently exists when \(h_{I}\) vanishes. Therefore, the antidark soliton and periodic wave are shown in forms of exponential and trigonometric functions, respectively. Specifically, the analytical expressions read as, for the soliton,

$$\begin{aligned} A_{S}^{[1]}=\left( a+n_{1}\frac{G_{S}^{[1]}+i H_{S}^{[1]}}{D_{S}^{[1]}}\right) e^{i \rho },\quad B_{S}^{[1]}=b, \end{aligned}$$
(2.5)

with

$$\begin{aligned} G_{S}^{[1]}= & {} k_{1} k_{2}\cosh (2 \chi _{I})-\cosh (x h_{I})\sin (2 \chi _{R}), \\ H_{S}^{[1]}= & {} \cos (2 \chi _{R})\sinh (x h_{I}),\\ D_{S}^{[1]}= & {} -k_{1} k_{2}\sin (2 \chi _{R})+\cosh (x h_{I})\cosh (2 \chi _{I}), \end{aligned}$$

and for the periodic wave,

$$\begin{aligned} A_{P}^{[1]}=\left( a+n_{1}\frac{G_{P}^{[1]}+i H_{P}^{[1]}}{D_{P}^{[1]}}\right) e^{i \rho } ,\quad B_{P}^{[1]}=b. \end{aligned}$$
(2.6)

with

$$\begin{aligned} G_{P}^{[1]}= & {} k_{1} k_{2} \cos (x h_{R})\cosh (2 \chi _{I})-\sin (2 \chi _{R}), \\ H_{P}^{[1]}= & {} k_{1} k_{2} \sin (x h_{R})\sinh (2 \chi _{I}),\\ D_{P}^{[1]}= & {} -k_{1} k_{2} \cos (x h_{R})\sin (2 \chi _{R})+\cosh (2 \chi _{I}). \end{aligned}$$

Figure 5 describes a soliton that does not propagate along x direction. It is shown that this soliton lies on a plane-wave background with the peak \((a\sqrt{\beta \gamma }-2 n)^{2}\). This kind of wave is referred to the antidark soliton which was firstly reported in the scalar NLS system with the third-order dispersion [44, 45]. Recent studies on the NLS-MB system have also presented the similar structures [35]. Further, for the value of a that becomes zero, this soliton will be converted into a standard bright soliton. Figure 6 exhibits the periodic wave with the period \(P=\frac{\pi }{h_{R}}\). It is interesting, as it looks like higher-order wave but appears from the same solution.

Fig. 8
figure 8

The effects of \(\alpha \), \(\beta \) and \(\gamma \) on the multi-peak solitons with \(a=2, \omega =1,k_{1}=1,k_{2}=-1\) and \(\lambda _{1}=\lambda _{2}^{*}=7.2-0.7\)i. a \(\alpha =-1,\beta =1\); b \(\alpha =-1,\gamma =1\); c \(\beta =1,\gamma =1\)

Full size image

In particular, as the period \(h_{R}\) is close to zero, namely \(n\rightarrow a\sqrt{\beta \gamma }\), the periodic wave will turn into the W-shaped soliton, as shown in Fig. 7. In this case, the solution (2.1) is transformed into

$$\begin{aligned} A_{RS}^{[1]}=-\frac{a e^{i x \omega } \left( a^2 \beta \gamma x^2-3\right) }{a^2 \beta \gamma x^2+1}. \end{aligned}$$
(2.7)

The maximum height (\(9a^{2}\)) of the W-shaped wave is nine times the background intensity, while the minimum is zero. By comparison with the W-shaped soliton in Fig. 2, the soliton in Fig. 7 has two different features: (1) It shows a single-peak structure without oscillating tails; (2) it has a rational expression.

We further discuss the effects of the coefficients \(\alpha \), \(\beta \) and \(\gamma \) on the transformed solitons. The coefficients \(\alpha \), \(\beta \) and \(\gamma \), which are related to the parameters \(c_{1}\), \(c_{2}\), \(l_{1}\) and \(l_{2}\), are defined by Eq. (1.3). Figure 8a, b indicates that the amplitudes of the multi-peak solitons decrease with increasing the values of \(\beta \) and \(\gamma \). In addition, increasing the values of \(\beta \) and \(\gamma \) leads to stronger localization and a smaller oscillation period for the multi-peak soliton. This means that we can control the amplitudes of the transformed nonlinear waves by adjusting the two group velocity coefficients \(c_{1}\) and \(c_{2}\) and the parameter \(l_{2}\) that reflects the interaction of the wave packet and the meanflow. However, as shown in Fig. 8c, changing the value of \(\alpha \) does not affect the characteristics of the multi-peak soliton.

3 Nonlinear wave interactions

In this section, we study the characteristics of interactions between the nonlinear waves presented above. Due to the diversity of the transformed waves (in fact, there are many kinds of nonlinear wave interactions), we only exhibit several typical nonlinear superposition patterns that derive from the two-breather solutions.

By virtue of the n-fold DT [30, 41], the two-breather solution of System (1.2) is given as

$$\begin{aligned} A^{[2]}_{B}=A^{[0]}+\frac{4 i}{\sqrt{\beta \gamma }}\frac{\varDelta _{1}^{[2]}}{\varDelta ^{[2]}}, \quad B^{[2]}_{B}=B^{[0]}-\frac{4 i}{\beta } \left( \frac{\varDelta _{2}^{[2]}}{\varDelta ^{[2]}}\right) _{t}, \end{aligned}$$
(3.1)

with

$$\begin{aligned}&A^{[0]}=a e^{i \rho } ,\quad B^{[0]}=b,\\&\lambda _{1}=\lambda _{2}^{*}=m_{1}+n_{1} i,\quad \lambda _{3}=\lambda _{4}^{*}=m_{2}+n_{2} i, \\&\psi _{2}=-\varphi _{1}^{*},\quad \varphi _{2}=\psi _{1}^{*};\quad \psi _{4}=-\varphi _{3}^{*},\quad \varphi _{4}=\psi _{3}^{*}; \\&\varphi _{j}={k_{1}}\frac{i h_{j}+2 i \lambda _{j}+i\omega }{a \sqrt{\beta \gamma }}e^{i (\varpi _{j}+\frac{\rho }{2})}+k_{2}e^{-i(\varpi _{j}- \frac{\rho }{2})},\\&\psi _{j}=k_{1} e^{i (\varpi _{j}-\frac{\rho }{2})}+k_{2} \frac{i h_{j}+2 i \lambda _{j}+i \omega }{a \sqrt{\beta \gamma }}e^{-i(\varpi _{j}+\frac{\rho }{2})},\\&j=1,3, \quad k_{1}=k_{2}=1, \\&\varDelta _{1}^{[2]}= \left| \begin{array}{llll} \lambda _{1}\varphi _{1}&{}\quad \varphi _{1}&{}\quad -\lambda _{1}^{2}\varphi _{1}&{}\quad \psi _{1} \\ \lambda _{2}\varphi _{2}&{}\quad \varphi _{2}&{}\quad -\lambda _{2}^{2}\varphi _{2}&{}\quad \psi _{2}\\ \lambda _{3}\varphi _{3}&{}\quad \varphi _{3}&{}\quad -\lambda _{3}^{2}\varphi _{3}&{}\quad \psi _{3}\\ \lambda _{4}\varphi _{4}&{}\quad \varphi _{4}&{}\quad -\lambda _{4}^{2}\varphi _{4}&{}\quad \psi _{4} \end{array}\right| , \\&\varDelta _{2}^{[2]}= \left| \begin{array}{llll} -\lambda _{1}^{2}\varphi _{1}&{}\quad \varphi _{1}&{}\quad \lambda _{1}\psi _{1}&{}\quad \psi _{1}\\ -\lambda _{2}^{2}\varphi _{2}&{}\quad \varphi _{2}&{}\quad \lambda _{2}\psi _{2}&{}\quad \psi _{2}\\ -\lambda _{3}^{2}\varphi _{3}&{}\quad \varphi _{3}&{}\quad \lambda _{3}\psi _{3}&{}\quad \psi _{3}\\ -\lambda _{4}^{2}\varphi _{4}&{}\quad \varphi _{4}&{}\quad \lambda _{4}\psi _{4}&{}\quad \psi _{4} \end{array}\right| ,\\&\varDelta ^{[2]}= \left| \begin{array}{llll} \lambda _{1}\varphi _{1}&{}\quad \varphi _{1}&{}\quad \lambda _{1}\psi _{1}&{}\quad \psi _{1} \\ \lambda _{2}\varphi _{2}&{}\quad \varphi _{2}&{}\quad \lambda _{2}\psi _{2}&{}\quad \psi _{2}\\ \lambda _{3}\varphi _{3}&{}\quad \varphi _{3}&{}\quad \lambda _{3}\psi _{3}&{}\quad \psi _{3}\\ \lambda _{4}\varphi _{4}&{}\quad \varphi _{4}&{}\quad \lambda _{4}\psi _{4}&{}\quad \psi _{4} \end{array}\right| . \end{aligned}$$
Fig. 9
figure 9

Elastic collision between two breathers with \(a=2, b=1, \gamma =4,\omega =1,\alpha =4,\beta =1,k_{1}=1,k_{2}=1, \lambda _{1}=\lambda _{2}^{*}=1.8i\) and \( \lambda _{3}=\lambda _{4}^{*}=-0.8+1.8i\)

Full size image
Fig. 10
figure 10

The interaction between two W-shaped solitons with \(a=2, b=1, \gamma =4,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=1, \lambda _{1}=\lambda _{2}^{*}=-0.5+2.1i\) and \( \lambda _{3}=\lambda _{4}^{*}=-0.5+2.2i.\) The nonlinear interaction forms an oscillation W-shaped soliton

Full size image
Fig. 11
figure 11

The interaction between two antidark solitons with \(a=2, b=1, \gamma =4,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=1, \lambda _{1}=\lambda _{2}^{*}=-0.5-2.2i\) and \( \lambda _{3}=\lambda _{4}^{*}=-0.5-2.3i.\) The nonlinear interaction forms a W-shaped soliton

Full size image

Figure 9 displays the elastic interaction between two breathers of System (1.2). It is found that the two-breather interaction produces a second-order central rogue wave in (x–t) plane. Similar to the case in Sect. 3, to convert the two-breather solutions into the two-soliton ones on constant backgrounds, the parameter b also needs to meet the transition condition (2.3), namely \(k=0\). For different kinds of nonlinear superpositions, we can choose the corresponding real (\(m_{j}\)) and imaginary (\(n_{j}\)) parts of eigenvalues (\(\lambda _{j}\)) in solution (3.1). The values of \(m_{j}\) and \(n_{j}\) (\(j=1, 3\)) control the types of the transformed nonlinear waves. In the following, we present six kinds of nonlinear superposition patterns.

3.1 The oscillation W-shaped soliton

We first consider the nonlinear superposition of two stationary W-shaped solitons, which is exhibited in Fig. 10. The two eigenvalues \(\lambda _{1}=-0.5+2.1i\) and \(\lambda _{3}=-0.5+2.2i\) meet the conditions of the W-shaped solitons. Interestingly, instead of a higher-order W-shaped soliton, these two localized waves form an oscillation W-shaped soliton with higher intensity (\(|A(0, 0)|^{2}=4(1+n_{1}+n_{2})^{2}\)). \(|A|^{2}\rightarrow a^{2}\) by assuming \(z \rightarrow \infty \), \(t \rightarrow \infty \) which gives the asymptotic plane. Due to the nonpropagating characteristic of the nonlinear waves, the nonlinear superposition could show some novel features. The interaction between two same type waves generates a different one.

3.2 The W-shaped soliton

Next, we demonstrate the nonlinear interaction between two antidark solitons in Fig. 11. The two eigenvalues \(\lambda _{1}=m_{1}+in_{1}\) and \(\lambda _{3}==m_{3}+in_{3}\) are set to be \(-0.5-2.2i\) and \(-0.5-2.3i\), respectively. Surprisingly, we find that the interaction between the antidark solitons does not produce a second-order antidark soliton, but instead generates a W-shaped soliton.

Fig. 12
figure 12

The interaction between an antidark soliton and a W-shaped soliton with \(a=2, b=1, \gamma =4,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=1, \lambda _{1}=\lambda _{2}^{*}=-0.5-2.1 i\) and \( \lambda _{3}=\lambda _{4}^{*}=-0.5+4i\)

Full size image
Fig. 13
figure 13

The interaction between an antidark soliton and a W-shaped soliton with \(a=2, b=1, \gamma =4,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=1, \lambda _{1}=\lambda _{2}^{*}=-0.5-2.7i\) and \( \lambda _{3}=\lambda _{4}^{*}=-0.5+2.699i\)

Full size image

3.3 The antidark soliton and W-shaped soliton pair

We then study the interaction between two different types of nonlinear waves, for example, the antidark soliton and a W-shaped soliton. The eigenvalue \(\lambda _{1}=-0.5-2.1i\) is for the antidark soliton while \(\lambda _{3}=-0.5+4i\) for the W-shaped one. As shown in Fig. 12, an intriguing phenomenon is that the interaction forms an antidark soliton. By adjusting the values of eigenvalues \(\lambda _{1}\) (still for antidark soliton) and \(\lambda _{3}\) (still for W-shaped soliton), we observe a W-shaped soliton pair in Fig. 13. Each component in the W-shaped soliton pair has the same intensity. In spite of the similar single wave as the seeds, the nonlinear interactions can produce different patterns due to the choices of different eigenvalues \(\lambda _{1}\) and \(\lambda _{3}\). This means the mechanism of nonlinear superposition is controlled by the two eigenvalues. A complete and analytical study of the control mechanism of such composite solutions is an issue that requires special investigation.

Fig. 14
figure 14

The interaction between an oscillation W-shaped soliton and an oscillation M-shaped soliton with \(a=2, b=1, \gamma =4,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=1, \lambda _{1}=\lambda _{2}^{*}=4.5-2.7i\) and \( \lambda _{3}=\lambda _{4}^{*}=5.5+5.2i\)

Full size image

3.4 The multi-peak solitons

Finally, we display some multi-peak structures. Figure 14 describes the nonlinear interactions between the oscillation W-shaped soliton and oscillation M-shaped soliton. The superposition leads to the formation of a three-peak soliton that corresponds to the case \(|A(0, 0)|^{2}_{xx}>0\). Using different eigenvalues, we plot a four-peak soliton (\(|A(0, 0)|^{2}_{xx}<0\)) also produced by the oscillation W-shaped soliton and oscillation M-shaped soliton, as depicted in Fig. 15. A higher-order multi-peak soliton, which is composed of two first-order multi-peak solitons, is plotted in Fig. 16.

Fig. 15
figure 15

The interaction between an oscillation W-shaped soliton and an oscillation M-shaped soliton with \(a=2, b=1, \gamma =4,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=1, \lambda _{1}=\lambda _{2}^{*}=6-3.05i\) and \( \lambda _{3}=\lambda _{4}^{*}=5+2i\)

Full size image
Fig. 16
figure 16

The interaction between two multi-peak solitons with \(a=2, b=1, \gamma =4,\omega =1,\alpha =-1,\beta =1,k_{1}=1,k_{2}=1, \lambda _{1}=\lambda _{2}^{*}=-19-5i\) and \( \lambda _{3}=\lambda _{4}^{*}=-20+2i\)

Full size image

4 MI characteristics

In this section, we reveal the explicit relation between the transition and the distribution characteristics of MI growth rate for System (1.2). Firstly, let us recall the results of MI analysis on System (1.2). As shown in Ref. [30], System (1.2) admits the following continuous-wave solutions,

$$\begin{aligned} {A}(x,t)= & {} ae^{i(\omega x+k t)},\nonumber \\ {B}(x,t)= & {} b, \end{aligned}$$
(4.1)

where \( a, \omega , k \) and b are real parameters. A perturbed nonlinear background can be expressed as

$$\begin{aligned} {A}(x,t)= & {} \left( a+\epsilon {\widehat{A}}(x,t)\right) e^{i(\omega x+kt)},\nonumber \\ {B}(x,t)= & {} b+\epsilon {\widehat{B}}(x,t). \end{aligned}$$
(4.2)

Taking Eq. (4.2) into System (1.2) yields the evolution equation for the perturbations as

$$\begin{aligned}&-a \beta {\widehat{B}}(x,t)+ik{\widehat{A}}^{(1,0)}(x,t)+i\omega {\widehat{A}}^{(0,1)}(x,t)\nonumber \\&\quad +{\widehat{A}}^{(1,1)}(x,t)=0,\nonumber \\&{\widehat{B}}^{(1,0)}(x,t)+\frac{1}{2}a \gamma {\widehat{A}}^{(0,1)}(x,t)\nonumber \\&\quad +\frac{1}{2}a \gamma {\widehat{A}}^{*(0,1)}(x,t)=0. \end{aligned}$$
(4.3)

Noting the linearity of Eq. (4.3) with respect to \(\widehat{A}\) and \(\widehat{B}\), we introduce

$$\begin{aligned} {\widehat{A}}(x,t)= & {} U\cos (\varLambda x-\varOmega t)+iV\sin (\varLambda x-\varOmega t),\nonumber \\ {\widehat{B}}(x,t)= & {} K\cos (\varLambda x-\varOmega t), \end{aligned}$$
(4.4)

which is characterized by the wave number \(\varLambda \) and frequency \(\varOmega \). Using Eq. (4.4) into Eq. (4.3) gives a linear homogeneous system of equations for U and K:

$$\begin{aligned}&-i a U \varLambda k+i a\omega U \varOmega +i a V \varLambda \varOmega =0, \end{aligned}$$
(4.5)
$$\begin{aligned}&-aV \varLambda k+a\omega V \varOmega -\frac{a^{2} U \beta \gamma \varOmega }{\varLambda }+aU \varLambda \varOmega =0.\nonumber \\ \end{aligned}$$
(4.6)

From the determinant of the coefficient matrix of Eqs. (4.5)–(4.6), the dispersion relation for the linearized disturbance can be determined as

$$\begin{aligned} \varOmega ^2 \left( -a \beta \gamma +\varLambda ^2-\omega ^2\right) -k^2 \varLambda ^2+2 k \varLambda \omega \varOmega =0. \end{aligned}$$
(4.7)

Solving the above equation, we have

$$\begin{aligned} \varOmega =\frac{k \varLambda \omega \pm \sqrt{k^2 \varLambda ^2 \left( \varLambda ^2-a \beta \gamma \right) }}{a \beta \gamma -\varLambda ^2+\omega ^2}. \end{aligned}$$
(4.8)

Figure 17 shows the characteristics of MI on the (\(\varLambda , k\)) plane. It is found that the MI exists in the region \(-a\sqrt{\beta \gamma }\) \(<\varLambda <a\sqrt{\beta \gamma }\). Reference [30] has found that the rogue-wave existence condition of System (1.2) is strictly related to baseband MI, namely the MI whose bandwidth includes arbitrary small frequencies. In this case, we obtain the parameter condition \(\beta \gamma >0\) [30]. Hereby, we discover that the MI growth rate distribution is symmetric with respect to

$$\begin{aligned} k=0, \end{aligned}$$
(4.9)

i.e.,

$$\begin{aligned} b=-\frac{\alpha }{\beta }. \end{aligned}$$
(4.10)

The \(k=0\) line (red dashed line) in Fig. 17 corresponds to a modulational stability (MS) region where the growth rate is vanishing in the low perturbation frequency region. More interestingly, one can find that the MS condition (4.9) [or (4.10)] is completely consistent with the condition (2.3) which converts breathers into stationary nonlinear waves. Our finding suggests that the transition between breathers and stationary nonlinear waves can occur in the MS region with the low-frequency perturbations.

Fig. 17
figure 17

Characteristics of MI growth rate \( \varOmega \) on \((\varLambda ,k)\) plane with \(a=1\), \(\gamma =4\), \(\beta =1\) and \(\omega =1\). Here the dashed red lines represent the stability region in the perturbation frequency region \(-a\sqrt{\beta \gamma }<\varLambda <a\sqrt{\beta \gamma } \) , which is given as \(b=-\frac{\alpha }{\beta }\). (Color figure online)

Full size image

5 Conclusions

In summary, we have investigated the transition between breathers and stationary nonlinear waves for System (1.2). Some intriguing different types of stationary nonlinear waves, including multi-peak solitons, M-shaped soliton, periodic wave, antidark soliton and W-shaped soliton, have been demonstrated graphically and analytically. The reasons and the conditions for transition have been presented explicitly. We have found that the real part of eigenvalue m has effects on the peak numbers of the multi-peak solitons, while the imaginary part n controls the transition between the oscillating W-shaped soliton and M-shaped soliton. Further, we have studied the interactions between those transformed nonlinear waves. Due to the nonpropagating characteristic, the nonlinear superposition of two types of nonlinear waves has exhibited some novel features (see Figs. 10, 11, 12, 13, 14, 15, 16). Finally, we have shown that this transition is strictly associated with the MI analysis that involves a MS region.