1 Introduction

Over the past several decades, solitons based on the nonlinear Schrödinger equation (NLSE) exhibit widespread applications in various branches of physics, engineering and other disciplines [1,2,3,4]. Spatial and spatiotemporal solitons reveal different types of localized modes, including fundamental solitons [5, 6], Peregrine solution and breather [7, 8], multipole solitons [9, 10] and vortex solitons [11, 12], and so on.

Vortex solitons are optical beams which exist phase singularities mixed within the wave front curvature and carry a nonzero angular momentum, and extensively investigated in the context of optical tweezers [13], entanglement states of photons [14] and trapping and guiding of cold atoms [15]. In many kinds of media, such as Kerr, saturable-atomic and photorefractive nonlinear media, optical vortices have been studied experimentally and theoretically [16, 17]. If a vortex generates an effective axisymmetric potential well, then this well may trap a bright two-dimensional solitary wave, producing some complex vector vortex solitons, such as the vortex-bright soliton [17], the half-quantum vortex [18] and the filled-core vortex [19].

In conservative nonlinear media, vortex solitons always lead to the symmetry breaking azimuthal instability and collapse into many fundamental solitons. This instability can also be inhibited. In graded-index optical fibers [20], nonlinear photonic crystals with defects and optical lattices with defect [21], the confined potentials can suppress the azimuthal instability of vortices. Although vortex soliton in the (3+1)-dimensional case has been studied in Bose–Einstein condensates (BEC) [22], the influence of the transverse modulation on vortex soliton in the (3+1)-dimensional spatially modulated cubic–quintic (CQ) nonlinear media is relatively few studied.

As we all know, the CQ nonlinearities are considered when the optical field frequency is close to a resonant frequency of the optical fiber material in nonlinear optics [23, 24]. On the other hand, in BEC with high density, the three-body interaction must be considered, and thus, the CQNLSE is used to describe this case [25]. After Serkin et al. firstly studied the topological quasi-soliton solutions of the variable coefficient CQNLSE [26], rich soliton modes of this equation have been derived [27, 28]. Recently, soliton solutions in the media with CQ nonlinearities modulated in space and time have also been discussed [29,30,31]. However, these works studied solitons based on (1+1)-dimensional and (2+1)-dimensional cases [28,29,30,31], (3+1)-dimensional vortex solitons in the spatially modulated CQ nonlinear media with the transverse modulation have not been reported. In this paper, we focus on this case.

2 Model and exact vortex soliton solutions

In nonlinear fiber optics, a general (3+1)-dimensional NLSE is written as [35, 36]

$$\begin{aligned} \text {i}\frac{\partial A}{\partial z}+\frac{1}{2\beta _0}\left( \frac{\partial ^2 A}{\partial x^2}+\frac{\partial ^2 A}{\partial y^2}\right) +\frac{\beta _2}{2}\frac{\partial ^2 A}{\partial t^2} +\gamma |A|^{2}A=0, \end{aligned}$$
(1)

where the pulse envelope \(A=A(z,x,y,t)\), the dispersion parameter \(\beta _2\) can be positive or negative with magnitude of the order of \(10^{-3}-10^{-2} \mathrm{ps^2 /m} \) [35] and the propagation parameter \(\beta _0 = 2\pi n_0 /\lambda \) with the refractive index \(n_0\) and the wave length of the beam \(\lambda \) [36]. The unit of the nonlinear parameter \(\gamma \) and \(| A|^2 \) is \(W^{-1} \mathrm{m}\) and \(W \mathrm{m^{-2}}\), respectively. By scaling, we define the following dimensionless variables \(x=x/\rho ,y=y/\rho ,t=t/\tau ,z=z/L_\mathrm{D},\phi =A\rho /\sqrt{P_0},p=\gamma P_0L_\mathrm{D}/\rho ^2\) [35] with lengths for dispersion or diffraction (\(L_\mathrm{DS}=L_\mathrm{DF}=L_\mathrm{D}\)), the radius of the beam \(\rho \), the timescale of the soliton \(\tau \), power \(P_0\), Eq. (1) can be transformed into the dimensionless form. Considering CQ nonlinearities, we obtain the following dimensionless NLS equations with self-focusing cubic and self-defocusing quintic nonlinearity [36, 37]

$$\begin{aligned} \text {i}\frac{\partial \phi }{\partial z}+\frac{1}{2}\left( \frac{\partial ^2 \phi }{\partial x^2}+\frac{\partial ^2 \phi }{\partial y^2}+\frac{\partial ^2 \phi }{\partial t^2}\right) +p|\phi |^{2}\phi -q|\phi |^{4}\phi =0, \end{aligned}$$
(2)

where \(\phi =\phi (z,x,y,t)\).

Recently, spatially modulated nonlinearity and transverse modulation are introduced to study the dynamics of solitons [29,30,31], and thus, the (3+1)-dimensional CQNLSE with spatially modulated nonlinearity and transverse modulation reads

$$\begin{aligned}&\text {i}\frac{\partial \phi }{\partial z}+\frac{1}{2}\left( \frac{\partial ^2 \phi }{\partial x^2}+\frac{\partial ^2 \phi }{\partial y^2}+\frac{\partial ^2 \phi }{\partial t^2}\right) \nonumber \\&\quad +\,\chi _3(r)|\phi |^{2}\phi +\chi _5(r)|\phi |^{4}\phi +R(r)\phi =0, \end{aligned}$$
(3)

where the cubic nonlinearity coefficient \(\chi _3(r)\), quintic nonlinearity coefficient \(\chi _5(r)\) and the transverse modulation R(r) are all functions of radial coordinate \(r\equiv (x,y,t)\). In BEC, \(\phi \) represents the macroscopic wave function of the condensate with time z, R(r) is the external potential. In this paper, we choose parabolic transverse modulation or external potential. Other external potentials can be found in Refs. [32,33,34].

In the following, we use dimensionless form and units to calculate, and these results can be easily converted to actual experimental units following the guidelines by considering some similar values in an experiment on spatiotemporal soliton in a planar glass waveguide, namely wave length \(\lambda = 1\,\mathrm{\mu m}, \beta _2= 10^{-2} \mathrm{ps^2 /m}\), and the timescale \(\tau = 60\) fs; thus, the propagation length \(L_\mathrm{D} = 36\) cm and the beam width \(\rho \approx 239\,\mathrm{\mu m}\).

We assume that Eq. (3) has the spatially localized vortex soliton solution in the form

$$\begin{aligned} \phi (r,\theta ,\varphi ,z)=\rho (r)\psi (\theta ,\varphi )\exp (-\text {i}\sigma z), \end{aligned}$$
(4)

where \(\sigma \) is the propagation constant in optics and the chemical potential in BEC, and the real function \(\rho (r)\) satisfies the localization condition \(\lim _{r\rightarrow \pm \infty }\rho (r)=0\).

Substituting Eq. (4) into Eq. (3), and considering \(\psi (\theta ,\varphi )=\sqrt{\frac{(2l+1)(l-k)!}{2\pi (l+k)!}}\) \(\times P_l^k(\cos \theta )\exp (\text {i}k\varphi )\) where the topological charge k and the associated Legendre polynomials \(P_l^k(\cos \theta )\) with the degree l and order k satisfying \(l\ge k\ge 0\), one obtains

$$\begin{aligned}&\frac{r^2}{\rho }\left\{ \frac{1}{r^2}\frac{\partial }{\partial r}(r^2\frac{\partial \rho }{\partial r})+ 2[\sigma +R(r)] \rho \right. \nonumber \\&\quad \left. +\,2\chi _3(r)\rho ^{3}+2\chi _5(r)\rho ^{4}\right\} =l(l+1), \end{aligned}$$
(5)

Following the scheme proposed in Refs. [28,29,30,31], if we assume \(\rho (r)\equiv \varrho (r)\varPsi [\zeta (r)]\) where \(\varPsi (\zeta )\) satisfying

$$\begin{aligned} \frac{d^{2}\varPsi }{d\zeta ^{2}}+G_1 \varPsi +G_{3}\varPsi ^{3}+G_{5}\varPsi ^{5}=0, \end{aligned}$$
(6)

with three constants \(G_1,G_3\) and \(G_{5}\) and consider \(\zeta (r)\equiv \int _{0}^{r}\varrho ^{-2} (s)s^{-1}\mathrm{d}s\), then Eq. (5) transforms into

$$\begin{aligned} \varrho ^{\prime \prime }+\frac{2}{r}\varrho ^{\prime }+\left[ 2\sigma +2R(r)-\frac{l(l+1)}{r^{2}}\right] \varrho =\frac{G_1}{r^{4}\varrho ^{3}}, \end{aligned}$$
(7)

and nonlinear functions \(\chi _3(r)\) and \(\chi _5(r)\) satisfy \(\chi _3(r)= G_{3}r^{-2}\varrho ^{-6}(r)/2,\chi _5(r)= G_{5}r^{-2}\varrho ^{-8}(r)/2\).

If we choose \(\varrho (r)=\eta (r)/r\), then Eq. (7) changes into the Ermakov–Pinney equation as \(\eta _{rr}+[2\sigma +2R(r)-l(l+1)/r^2]\eta =G_1/\eta ^3\) [38]. If \(G_1\) is a constant, then \(\eta (r)\) can be expressed as \( \eta =\sqrt{\alpha \xi _1^2+2\beta \xi _1\xi _2+\gamma \xi _2^2}\) where \(G_1=(\alpha \gamma -\beta ^2)W^2\) with three constants \(\alpha , \beta , \gamma \) and Wronskian \(W=\xi _1\xi _{2r}-\xi _2\xi _{1r}=\text {constant}\) with \(\xi _1(r)\) and \(\xi _2(r)\) being two linearly independent solutions of \(\xi _{rr}+[2\sigma +2R(r)-l(l+1)/r^2]\xi =0\). Further, if \(G_1=0\) and R(r) is the parabolic transverse modulation with \(R(r)=\frac{1}{2}\omega ^2r^{2}\), then \(\varrho (r)=r^{-\frac{3}{2}}[c_{1}\mathrm{M}(\frac{\sigma }{2\omega },\frac{l}{2}+\frac{1}{4},\omega r^{2})+c_{2}\mathrm{U}(\frac{\sigma }{2\omega },\frac{l}{2}+\frac{1}{4},\omega r^{2})]\) with the Whittaker’s \(\mathrm{M}\) and \(\mathrm{U}\) functions [39], and constants \(c_{1}c_{2}>0\). Without considering the transverse modulation with \(R(r)=0\), \(\varrho (r)=r^{-\frac{1}{2}}[c_{3}J(l+1/2,\sqrt{2\sigma }r)+c_{4}{Y}(l+1/2,\sqrt{ 2\sigma }r)]\) with the Bessel functions of the first kind J and second kind Y, and constants \(c_{3}c_{4}>0\).

Using a one-to-one correspondence between Eq. (3) and Eq. (6) with \(G_1=0,G_3=\frac{2}{3}\mu n^2K^2(m)(m^4-m^2+1),G_5=\frac{1}{9}\mu ^2 n^2K^2(m)(m^2-2)(2m^2-1)(m^2+1)\), we obtain two families of exact vortex soliton solutions for Eq. (3) as

$$\begin{aligned} \phi _{1n}= & {} \sqrt{\frac{(2l+1)(l-k)!}{2\pi (l+k)!}}P_l^k\nonumber \\&(\cos \theta )\frac{3\varrho (r) \text {sn}[\lambda K(m),m]}{\sqrt{3\mu [3-(m^2+1)\text {sn}^2[\lambda K(m),m]]}}e^{\text {i}k\varphi -\text {i}\sigma z},\nonumber \\ \end{aligned}$$
(8)

with \(\lambda =n\zeta +1\) for \(n=1,3,5,\ldots ,\) while for \(n=2,4,6,\ldots ,\)

$$\begin{aligned} \phi _{2n}= & {} \sqrt{\frac{(2l+1)(l-k)!}{2\pi (l+k)!}}P_l^k\nonumber \\&(\cos \theta )\frac{3\varrho (r) \text {sn}[nK(m)\zeta ,m]}{\sqrt{3\mu [3-(m^2+1)\text {sn}^2[nK(m) \zeta ,m]]}}e^{\text {i}k\varphi -\text {i}\sigma z},\nonumber \\ \end{aligned}$$
(9)

where \(\mu \) is a real constant, the integer n is associated with the soliton order number, sn is the Jacobian elliptic sine function with the modulus m and \(K(m)=\int ^{\pi /2}_0[1-m^2\sin ^2(\xi )]^{-1/2}\mathrm{d}\xi \) is elliptic integral of the first kind [40].

Fig. 1
figure 1

(Color online) Vortex solitons with (a, c, e, g, i, k, m, o, q) solution (8) and (b, d, f, h, j, l, n, p, r) solution (9) for the disappearing transverse modulation \(R=0\). The parameters are chosen as \(\sigma =0.3,\) \(\mu =0.1,m=0.1,c_{3}=1,c_4=1.5,n=1\) with a, b \(k=l=0\); c, d \(k=0,l=1\); e, f \(k=l=1\); g, h \(k=0,l=2\); i, j \(k=1,l=2\); k, l \(k=l=2\); m, n \(k=1,l=3\); o, p \(k=2,l=4\) and q, r \(k=3,l=5\)

Full size image
Fig. 2
figure 2

(Color online) Vortex solitons with solution (8) for the disappearing transverse modulation \(R=0\). The parameters are chosen as those in Fig. 1 except for a \(k=0,l=1,n=3\), b \(k=l=1,n=7\) and c \(k=2,l=4,n=3\)

Full size image

3 Construction of vortex solitons

Solutions (8) and (9) describe vortex solitons with different configurations. For the disappearing transverse modulation \(R=0\), vortex solitons of solutions (8) and (9) with different values of k and l are shown in Fig. 1. When \(k=l=0\), localized spheres of solutions (8) and (9) are displayed in Fig. 1a, b, respectively. When \(k=0,l=1\), solution (9) describes a cylinder as shown in Fig. 1d, and solution (8) describes a sphere surrounded by a cylinder as shown in Fig. 1c. When \(k=l=1\), two discs with symmetrical distribution on two sides of the plane \(t=0\) are constructed in Fig. 1f, and two symmetrical spheres surrounded by two discs are constructed in Fig. 1e. In Fig. 1g for solution (8) with \(k=0,l=2\), a pair of drip-shaped structures embed above and below the torus-shaped structure in the middle, and the whole structure is surrounded by a cylinder, which also appears in Fig. 1h for solution (9). When \(k=1,l=2\), two cones with symmetrical distribution on two sides of the plane \(t=0\) are constructed in Fig. 1j, and two symmetrical toruses surrounded by two cones are constructed in Fig. 1i. When \(k=l=2\), structures in Fig. 1k, l are similar to those in Fig. 1e, f, and the difference is that the distance between two discs adds. In Fig. 1m for solution (8) with \(k=1,l=3\), a pair of drip-shaped structures also appear above and below the pair of torus-shaped structures in the middle, and the whole structure is surrounded by two cones, where every cone is closed by a disc at the end. These cones also are shown in Fig. 1n for solution (9). When \(k=2,l=4\), structures in Fig. 1o, p are similar to those in Fig. 1m, n, and the difference is that all parts of structures enlarge. When \(k=3,l=5\), two nested structures with the arranges of two cones closed by a disc at the end one by one distribute symmetrically on two sides of the plane \(t=0\) in Fig. 1r, and inside the whole structure in Fig. 1r, a pair of drip-shaped structures also exist above and below the pair of torus-shaped structures in the middle in Fig. 1q. From these structures in Fig. 1e, f, k, l, and Fig. 1m–p, we find that similar structures can be constructed for the same value of \(l-k\).

Figure 2 displays multi-layer structures of vortex solitons for \(n=3,7\). Compared Fig. 1c for \(n=1\) with Fig. 2a for \(n=3\) and Fig. 1o for \(n=1\) with Fig. 2c for \(n=3\), there is an extra layer structure in both structures for \(n=3\). Compared Fig. 1e for \(n=1\) with Fig. 2b for \(n=7\), there are three extra layers in the structure for \(n=7\). Therefore, the number of the inner layer structure getting rid of the package covering layer is related to \((n-1)/2+1\) with the soliton order number n.

Fig. 3
figure 3

(Color online) Phases of vortex solitons with solutions (8) and (9) for the disappearing transverse modulation \(R=0\) at \(t=0\). The parameters are chosen as those in Fig. 1 except for a \(k=l=1,n=1\), b \(k=l=1,n=2\), c \(k=l=2,n=1\), d \(k=l=2,n=2\), e \(k=l=3,n=1\), f \(k=l=3,n=2\), g \(k=l=1,n=5\), h \(k=l=2,n=3\) and i \(k=3,l=5,n=3\)

Full size image

Figure 3 shows phases of vortex solitons with solutions (8) and (9) for the disappearing transverse modulation \(R=0\) at \(t=0\) when k, l, n are chosen as different values. All phases of vortex solitons in Fig. 3 exist an azimuthal jump around their cores with different values of k, l, n. In Fig. 3a, there exist two azimuthal jumps, and every jump realizes the change in \(2\pi \); however, the phase change from the first jump to second jump is \(\pi \). In Fig. 3b, there exist three azimuthal jumps with \(2\pi \) phase change in every jump, and two jumps one after another realize the change in \(\pi \); thus, two changes totally produce the change in \(2\pi \). In Fig. 3c, there exist six jumps, that is, five change in two jumps one after another, and thus, totally produces the change in \(5\pi \). Similar analysis for Figs. 3d–i can be used, and we omit it for the limit of length.

Fig. 4
figure 4

(Color online) a–d Intensity and e–f phase of vortex solitons in the presence of the parabolic transverse modulation \(R=\omega ^2 r^{2}/2\). The parameters are chosen as those in Fig. 1 except for \(\omega =0.01\) with a \(k=0,l=1,n=1\), b \(k=0,l=1,n=2\), c, e \(k=l=1,n=1\); d \(k=l=1,n=2\), f \(k=l=2,n=1\), g \(k=l=3,n=1\) and h \(k=l=4,n=1\)

Full size image

In the presence of the parabolic transverse modulation \(R=\varpi ^2 r^{2}/2\), vortex solitons can be also constructed. Figure 4 presents some cases of vortex solitons with the intensity and corresponding phases. When \(k=0,l=1\), solution (9) with \(n=1\) describes a cylinder with bell mouthed shape in Fig. 4b, and solution (8) with \(n=2\) describes a torus-shaped structure surrounded by a cylinder with bell mouthed shape in Fig. 4a. When \(k=l=1\), two trays with symmetrical distribution on two sides of the plane \(t=0\) are constructed in Fig. 4d, and two symmetrical flattened ellipsoids surrounded by two trays are constructed in Fig. 4c. From the phase plot in Fig. 4e, when \(m=l=1\), it is similar to the case for the disappearing transverse modulation \(R=0\), that is, there exist two azimuthal jumps with \(2\pi \) phase change in every jump, and two jumps one after another realize the change in \(\pi \). However, when \(k=l=2,3,4\), the phase change in vortex soliton in presence of the parabolic transverse modulation \(R=\varpi ^2 r^{2}/2\) is different from those for the disappearing transverse modulation \(R=0\). From Fig. 4f–h, all phases of vortex soliton exist k-jump with different values of the topological charge k, and every jump realizes the change in \(2\pi /k\); thus, k-jumps totally realize the azimuthal change in \(2\pi \) around their cores.

4 Conclusion

In conclusion, with the help of the variable separation principle with the similarity transformation, we derive two families of vortex soliton solution in the spatially modulated CQ nonlinear media governed by a (3+1)-dimensional CQNLSE with spatially modulated nonlinearity and transverse modulation. For the disappearing and parabolic transverse modulations, vortex solitons with different configurations are constructed. The similar configurations of vortex solitons exist for the same value of \(l-k\) with the topological charge k and degree number l. Moreover, the number of the inner layer structure of vortex solitons getting rid of the package covering layer is related to \((n-1)/2+1\) with the soliton order number n. For the disappearing transverse modulation, there exist phase azimuthal jumps around their cores of vortex solitons with \(2\pi \) phase change in every jump, and any two jumps one after another realize the change in \(\pi \). For the parabolic transverse modulation, all phases of vortex solitons exist k-jump with different values of the topological charge k, and every jump realizes the change in \(2\pi /k\); thus, k-jumps totally realize the azimuthal change in \(2\pi \) around their cores.