1 Introduction

Graphene is a two-dimensional (2D) allotrope of carbon in a hexagonal lattice which was realized and characterized by Novoselov et al. (2004). In spite of one atom thickness of graphene, it shows the optical absorption of 2.3% per layer which is practically independent of wavelength (Nair et al. 2008; Li et al. 2008). The unique optical and electronic properties of graphene such as high charge carrier mobility (experimentally measured at room temperature as high as \(15,000\;{\text{cm}}^{2} {\text{v}}^{ - 1} {\text{s}}^{ - 1}\)) (Geim and Novoselov 2007; Bolotin et al. 2008), thermal conductivity (Balandin et al. 2008), strong light-matter interactions, stability, high-speed operation, and gate-variable optical conductivity have made it an appropriate candidate to utilize in nanoscale devices (Novoselov et al. 2012; Geim 2009). Subwavelength plasmonic waveguides (Christensen et al. 2011; Ju et al. 2011; Jablan et al. 2009), modulators (Bao and Loh 2012; Gosciniak and Tan 2013), photodetectors (Liu et al. 2014), ultrafast lasers (Sun et al. 2010) are some of the graphene applications in the visible to terahertz ranges. Graphene devices have a great potential as a platform for broadband photonic integrated circuits.

Furthermore, the capability of tuning fermi level and controlling permittivity in graphene by chemical doping or gate voltage, has made it possible for nonlinear optics applications. Despite the fact that there are limitations in utilizing the nonlinear optical features of graphene such as high loss and small interaction volume, very high nonlinearity of graphene has been calculated in visible and near-infrared frequencies (Khurgin 2014; Hendry et al. 2010). At higher frequencies, the inter-band optical transitions occur at all photon energies which lead to strong nonlinear behavior of graphene. A third-order susceptibility of a single graphene layer was determined about \(\left| {\chi_{gr}^{3} } \right| = 1.5 \times 10^{ - 7} \;{\text{esu}}\) (electrostatic units) by Hendry et al.(2010).

In order to provide sub-wavelength field confinement and overcome the diffraction limit, plasmonic based structures are studied owing to their ability to support the surface-plasmon polariton (SPP) surface waves (Hayashi and Okamoto 2012; Oulton 2008; Dai et al. 2011). Surface plasmons polaritons are oscillation of localized free electrons along the interface between two materials with different permittivity such as dielectric–metal interface. On the other hand, graphene substantial characteristics in comparison with noble metals bring about extreme confinement, low loss and long propagation lengths of SPP waves in plasmonic devices (Liu et al. 2011; Ooi et al. 2013; Ishikawa et al. 2009; Gu et al. 2013). The propagation of SPPs on graphene is strongly dependent on the chemical potential. The plasmon loss in graphene becomes very small for a sufficiently high Fermi level. Various waveguide structures based on graphene layers have been proposed over the last decade including graphene–metal structure, graphene-based hybrid plasmonic waveguides, ribbon structures, negative permeability metamaterial and multi-layer graphene waveguides (Christensen et al. 2011; Ishikawa et al. 2009; Gu et al. 2013; Lu et al. 2013; Koester and Li 2012). The theoretical analysis of a plasmonic dielectric-loaded surface plasmon polariton (DLSPP) waveguide configuration with a double-layer graphene placed between a gold stripe and a dielectric ridge were studied and compared to silicon waveguides (Gosciniak and Tan 2013). Furthermore, a graphene-based hybrid plasmonic waveguide (GHPW) structure including two identical cylinder robs were proposed and the propagation length of 26.7 mm was achieved at 10 THz (Chen et al. 2016).

In the current paper, two different hybrid waveguides based on graphene with capability of subwavelength confinement is presented in telecommunication wavelength range and designed for nonlinear applications including frequency conversion processes, such as second and third harmonic generation, Q-switching and mode locking, quantum optics and so forth. The Sect. 2 discuss around the optical features of graphene and the conductivity and equivalent complex permittivity of graphene as a function of chemical potential is calculated numerically. The structure of photonic and plasmonic hybrid waveguides with an embedded graphene layer are theoretically studied. In order to achieve large nonlinear effects, the field confinement inside the LiNbO3 layer is discussed. Furthermore, the influence of geometric parameters, the refractive index of the gap layer and the chemical potential of the graphene on the modal area, propagation loss, and propagation length as well as mode effective index are separately investigated in Sects. 3 and 4. The performance of the presented waveguide structures as well as the results are analyzed subsequently.

2 Graphene optical properties

The complex surface conductivity of monolayer graphene is described by the Kubo formula (Ando et al. 2002) which has a direct relation with angular frequency (\(\omega\)), chemical potential (\(\mu_{c}\)), temperature (\({\text{T}}\)) and scattering rate (\(\Gamma\)). In the following equations, the optical response of graphene divided into two inter-band and intra-band transitions terms (\(\sigma_{g} = \sigma_{intra} + \sigma_{inter}\)).

$$\sigma_{intra} = i\frac{{e^{2} K_{B} T}}{{\pi \hbar^{2} \left( {\omega + i2\Gamma } \right)}}\left[ {\frac{{\mu_{c} }}{{K_{B} T}} + 2\ln \left( {\exp \left( { - \frac{{\mu_{c} }}{{K_{B} T}}} \right) + 1} \right)} \right]$$
$$\sigma_{inter} = i\frac{{e^{2} }}{{4\pi \hbar^{2} }}{ \ln }\left[ {\frac{{2\left| {\mu_{c} } \right| - \hbar \left( {\omega + i2\Gamma } \right)}}{{2\left| {\mu_{c} } \right| + \hbar \left( {\omega + i2\Gamma } \right)}}} \right]$$

where \(e\) is the electron charge, \(K_{B}\) is the Boltzmann constant, \(\hbar\) is the reduced Plank constant. The chemical potential in graphene corresponds to carrier concentration which can be tuned by an applied voltage (Wang 2008; Gao et al. 2012) of \(\mu_{c} = \tau (e\vartheta_{f}^{2} )\text{ / }\mu = \hbar \vartheta_{f} \left( {\pi n} \right)^{{1\text{ / }2}}\) where \(n\) is the carrier concentration, \(\vartheta_{f} \left( { \approx 10^{6} \;{\text{m/s}}} \right)\), is the Fermi velocity, \(\tau\) represents the electron relaxation time of graphene, which corresponds to the scattering parameter \(\tau^{ - 1} = 2\Gamma\) and \(\mu\) shows the carrier mobility. In the above equations we assume the scattering parameter of graphene to be \(\hbar\Gamma = 5\;{\text{meV}}\).

According to dependency of the waveguide’s performance on the permittivity of constituent materials, it’s crucial to determine the exact permittivity of graphene layer through the simulation. Furthermore, to enhance the interaction of multilayer graphene with adjacent layers and optimize the propagation parameters, the relative complex permittivity of graphene is derived from its 2D conductivity, \(\varepsilon_{g} = 1 + \frac{{i\sigma_{g} \eta_{0} }}{{\kappa_{0}\Delta }}\), here \(\Delta\) is the thickness of a graphene sheet, \(\eta_{0} \left( { \approx 377 \Omega } \right)\) is the impedance of air and \(k_{0} \left( { = 2\pi \text{ / }\lambda } \right)\) is the wavenumber of incident light wavelength.

Figure 1 depicts the conductivity in monolayer graphene, the complex relative permittivity, the intra-band permittivity and the inter-band permittivity as a function of different quantities of chemical potential at room temperature (300 K) for telecommunication wavelength \(\left( {1.55\;\upmu{\text{m}}} \right)\). The graphene thickness is set to \(0.35 \;{\text{nm}}\).

Fig. 1
figure 1

Calculated graphene’s complex a conductivity b relative permittivity c intra-band permittivity d inter-band permittivity as a function of chemical potential, λ = 1550 nm, T = 300 K

According to transitions that originate from states near the Fermi surface (\(2\left| {\mu_{c} } \right| = \hbar \omega\)), the band-filling effect dominates at photon energy (\(\hbar \omega = 800 \;{\text{meV}}\)). Owing to the Pauli blocking phenomenon in graphene, the real part of the conductivity drops substantially close to the \(0.4 \;{\text{eV}}\) of chemical potential while the imaginary part starts to increase (Fig. 1a). At the point where \(\mu_{c}\) reaches \(0.492 \;{\text{eV}}\) the absolute value of graphene’s conductivity and permittivity reduce to near zero with complex permittivity of \((\varepsilon_{g} = 0.024 + 0.233{\text{i}})\), so when \(\mu_{c} < 0.492 \;{\text{eV}}\) inter-band transitions dominate the absorption channel in the mid, near-infrared and visible spectral range. (Figure 1d) and graphene behaves like a dielectric material.

At the interval \(3\hbar \omega /5 < \mu_{c} < \hbar \omega /2\) graphene is transparent but optical confinement and wave conducting behavior is poor (Mikhailov and Ziegler 2007). By increasing the chemical potential over the \(0.492\; {\text{eV}}\), graphene optical transitions tend to occur within the band which causes the intra-band transitions to be dominant (Fig. 1c). So, the metallic feature of graphene with capability of exciting Surface Plasmon Polaritons appears gradually. At longer wavelengths (far-infrared to terahertz) graphene operates like a thin metal which can be a proper alternative for noble metals.

Practically, the special properties of graphene are variable with the thickness and the number of graphene layers. Therefore, it’s crucial to determine the precise and possible thickness of graphene based on substrate material. In order to improve the performance of graphene-based devices, several studies of graphene thickness measurement in the range of \(0.3 \;{\text{nm}}\) to \(1 \;{\text{nm}}\) have been reported by optical microscopy and Raman spectroscopy (Berciaud et al. 2008; Casiraghi et al. 2007; Roddaro et al. 2007; Cao et al. 2011). On the other hand, the mobility increases with carrier density in multi-layer graphene indicating the presence of layer to layer coupling in graphene.

3 The photonic graphene-based waveguide

In order to optimize the performance of graphene waveguides and extend the application of such structures in integrated devices, the optical and geometrical characteristics of several configurations are studied. Besides, the presented waveguides take advantage of Lithium Niobate’s (LiNbO3) large \(\chi^{\left( 2 \right)}\) coefficient and wide transmission window from UV to mid-IR (\(350 \;{\text{nm }}\;\;{\text{to}}\;\; 4.5\;\upmu{\text{m}}\)) (Nikogosyan 2006). The cross-section schematic and electric field distributions of the graphene-based integrated hybrid waveguide is illustrated in Fig. 2a. The waveguide consists of a LiNbO3 layer as a high index dielectric (\(n = 2.1\)) on top, a low index Polymethyl methacrylate (PMMA) layer (\(n = 1.48\)) (Beadie 2015) and a single graphene layer placed on the silicon dioxide (\(n = 1.54\)) substrate.

Fig. 2
figure 2

Cross section of a four-layer a hybrid graphene-based waveguide, consists of a lithium niobate layer (width w, height h), PMMA (thickness d), a graphene sheet (thickness t) and SiO2 as a substrate b electric mode area with a PMMA gap (h = 300 nm), c electric mode area with a PMMA gap (h = 700 nm), d electric mode area with a silicon gap (h = 300 nm), e electric mode area with a silicon gap (h = 700 nm)

The TE mode electric field intensity inside the waveguide with a low index gap layer (PMMA) is shown in Fig. 2b, c. Here the mode confinement improves by increasing the LiNbO3 thickness to 700 nm while the field tends to be localized within the gap region with the 300 nm height. Subsequently, to reach an efficient index contrast, a high index silicon (\(n = 3.48\)) are replaced with PMMA and the dependence of the mode area on the layer thickness is depicted in Fig. 2d, e. The electric field intensity are mostly confined in the LiNbO3 and a small part of the silicon layer interacts with light, where h = 700 nm.

To better understand the influence of the LiNbO3 layer dimensions on the propagating mode, the effective refractive index (\(Re\left( {\eta_{eff} } \right)\)) and the propagation loss (defined by \(Im\left( {\eta_{eff} } \right)\)) of the selected mode are illustrated in Fig. 4 as a function of the LiNbO3 layer thickness. The gap layer thickness (d) is fixed on 100 nm and a 0.35-nm-thick graphene is assumed for the photonic waveguide configuration (Fig. 2a).The chemical potential (\(\mu\)) and the scattering parameter (\(\hbar\Gamma\)) are set on \(0.6 \;{\text{eV}}\) and \(5 \;{\text{meV}}\), respectively. As it is shown in Fig. 3a, the effective index keeps rising for both low (PMMA) and high (Silicon) refractive index gap layer while the LiNbO3 thickness increases up to 700 nm. It should be mentioned that the proper refractive index contrast between PMMA and LiNbO3 layers brings about a great mode confinement and a high effective index at h = 700 nm. Further, the propagation loss in SiO2–Graphene–Silicon–LiNbO3 (S–G–S–L) structure decreases gradually by increasing the LiNbO3 height, so the imaginary part of effective index reaches to its lowest value of \(8.6 \times 10^{ - 6}\) at 700 nm. While the structure SiO2–Graphene–PMMA–LiNbO3 (S–G–P–L) has a gentle reduction in h > 600 nm (Fig. 3b).

Fig. 3
figure 3

a The effective refractive index [real (Neff)] and b the propagation loss [imaginary (Neff)] as a function of LiNbO3 thickness for two different structure, where w = 700 nm, d = 100 nm, t = 0.35 nm

The optical properties of the propagating mode in the waveguides can be changed by adjusting the graphene’s chemical potential induced by gate voltage. As a result, the charge-carrier density in the graphene is changed and accordingly the Fermi energy, EF is shifted. However, graphene’s chemical potential is highly susceptible to the substrate and the transfer processes which might create additional defects and undesirable impurities, leading to reduced electron mobility (Pirkle et al. 2011; Chen 2015). Therefore, the effect of the graphene’s chemical potential variations (\(0\;{\text{eV}} < \mu_{c} < 1\;{\text{eV}}\)) on the effective refractive index and the propagation loss are displayed in Fig. 4.

Fig. 4
figure 4

The effective refractive index [real (Neff)] and the propagation loss [imaginary (Neff)] as a function of chemical potential (μc) for the a SiO2–Graphene–PMMA–LiNbO3 and b SiO2–Graphene–Silicon–LiNbO3 structures, where w = 700 nm, h = 700 nm, d = 100 nm, t = 0.35 nm

The fluctuating behavior of the \(Re(N_{eff}\)) indicates the high dependency of the guided mode on the graphene’s phase while it changes from a dielectric material to a metallic graphene by increasing the chemical potential. In spite of the increase in the chemical potential until \(\mu_{c} < 0.4\;{\text{eV}}\), the propagation loss has an almost stable amount, while a substantial drop from \(\mu_{c} < 0.37\;{\text{eV}}\) to \(\mu_{c} < 0.46\;{\text{eV}}\) is observed. So at the same time, the \(Img(N_{eff}\)) of the propagating mode in the S–G–P–L waveguide (Fig. 4a), decreases sharply from the highest value \(4.2 \times 10^{ - 4}\) to \(9.8 \times 10^{ - 6}\) whereas, it reduces from \(7.3 \times 10^{ - 4}\) to \(1.7 \times 10^{ - 5}\) in the S–G–S–L structure (Fig. 4b). By approaching the chemical potential to \(\mu_{c} = 0.493\;{\text{eV}}\) (the point where the permittivity of the graphene is close to zero), an abrupt rise in the propagation loss is seen due to the lossy graphene medium. The further increase of the chemical potential over the \(0.493\;{\text{eV}}\) decreases the \(Img(\eta_{eff}\)) to the minimum amount \(4 \times 10^{ - 6}\) and \(7.9 \times 10^{ - 6}\) for the S–G–P–L and S–G–S–L structures, respectively.

4 The plasmonic graphene-based waveguide

In the current section, the characteristics of the propagating mode in a graphene integrated hybrid plasmonic waveguide are analyzed and the effect of the ridge dimensions as well as the graphene conductivity on the waveguide performance are studied. One of the main advantages of such structures is the presence of a metal layer which allows the waveguide to both support the SPPs and have gate contact at the same time.

According to the Fig. 5a, the waveguide consists of a LiNbO3 (\(n = 2.1\)) layer as the dielectric ridge, a silver strip (\(\varepsilon = - 130 + 3.32{\text{i}}\)) to support the surface plasmons and a very thin graphene sheet (0.35 nm) placed between the silver and the LiNbO3 layers. The layers rest on a silicon dioxide (\(n = 1.54\)) substrate and the whole structure is surrounded by air. Figure 5 shows the electric field intensity of the TM mode within the waveguide for two different values of LiNbO3 layer thickness. It is seen that the field is mainly localized inside the ridge at h = 500 nm which leads to a low propagation loss, while it tends to be distributed throughout the SiO2 and LiNbO3 layers at h = 200 nm.

Fig. 5
figure 5

Cross section of the a plasmonic graphene-based waveguide, consists of a LiNbO3 layer (w = 500 nm, h = 500 nm), a Silver strip (g = 30 nm), a graphene sheet (t = 0.35 nm) and SiO2 layer as a substrate b electric field distribution for the TM mode (h = 200 nm, µ = 0.6 eV), c electric field distribution for the TM mode (h = 500 nm, µ = 0.6 eV), d the electric field intensity inside the LiNbO3 layer (µ = 0.493 eV), e the electric field intensity inside the LiNbO3 layer (µ = 0. 5 eV)

In the proposed configuration, the modal intensity is focuesed near to the boundary between the silver and the dielectric ridge. So, in this situation, the graphene’s conductivity has a significant role in propagating modes inside the ridge and the modal confinement can be improved over a narrow chemical potential range between μ = 0.495 eV and μ = 0.52 eV, Fig. 5d, e.

To have a clear understanding of the supported SPP mode characteristics in the waveguide, the effective mode index (\(Re(N_{eff}\))) and the propagation length (\(L_{P}\)) for variations of the chemical potential (\(\mu\)) and the ridge thickness (h) are analyzed. The propagation length is defined as the distance that the electric field intensity of SPP reduces to (1/e) of its initial value and obtained from the imaginary part of the mode effective index \(L_{P} = \lambda /\left\{ {4\pi \times Im\left( {N_{eff} } \right)} \right\}\) where \(\lambda = 1.55 \;\upmu{\text{m}}\). It can be observed that the mode effective index has a direct relation with the ridge thickness variations. So in this way, \(Re(N_{eff}\)) changes from 1.49 to 1.8 while the ridge thickness increases from 200 nm to 500 nm. The smallest propagation length is achieved at h = 250 nm where the field begins to localized inside the LiNbO3 layer with low intensity. As the ridge thickness increases (h > 250 nm), more field extends into the ridge region causes the propagation length to reach around 38 µm at h = 500 nm. For the smaller ridge dimensions, h < 250 nm, the modal field cannot be confined completely within the ridge, Fig. 6a.

Fig. 6
figure 6

Variations of the mode effective index and the propagation length as a function of a the ridge thickness at µ = 0. 6 eV b chemical potential for h = 500 nm at 1.55 µm

Figure 6b shows that the propagation length changes slightly until μc = 0.39 eV, but a significant increase in the propagation length is observed when the μc reaches 0.42 eV. By adding the chemical potential, the optical absorption of graphene is increased and consequently the maximum propagation loss is found at μc = 0.494 eV. Further increase of the chemical potential above 0.494 eV, enhances the metalic feature of the graphene and improves the propagation of SPPs along the ridge-silver interface. In addition, the effective index of the propagating mode changes from the minimum quantity to its maximum value over a very narrow variations of chemical potential from 0.491 eV to 0.5 eV.

5 Conclusion

In order to analyze the incorporation of graphene to hybrid waveguides, the optical properties of graphene was studied. Next, two different types of graphene-based hybrid photonic and plasmonic waveguides were proposed at 1.55 μm. The performance of the presented waveguides were investigated in terms of the nonlinear material geometry, the field confinement inside the nonlinear material (LiNbO3) and the conductivity of graphene. It was represented that the propagation properties and the effective index of the guided mode can be controlled by tuning the graphene’s chemical potential induced by gate voltage. Moreover, it was realized that the propagation loss rises to the maximum value at μc = 0.493 eV and μc = 0.494 eV for the photonic and plosmonic waveguides, respectively.