1 Introduction

The large quantities of emerging media and devices lead to massive data volumes and high data rate requirements. According to [1], one-thousand-times greater capacity will be needed in the next decade. Table 1 shows the enhancement of the next-generation capabilities from IMT-2020 compared with IMT-Advanced [2]. The area traffic capacity and peak data rate need to be increased dozens of times. The most direct method to satisfy these demands is to increase the spectral efficiency and the available bandwidth.

Table 1 Enhancement of capabilities from IMT-Advanced to IMT-2020

Millimeter wave (mmWave) communication can fulfill the demand for a large bandwidth. In the mmWave band, GHz of bandwidth is available, and a rate of 4.5 Gbps can be realized in outdoor environments [3]. In addition, the short wavelength of mmWave enables large antenna arrays to be packed into a compact size. A narrow beamwidth is also easier to obtain through the mmWave band. However, the deficiency of mmWave is clear based on the Friis transmission formula,

$$ {P_{\mathrm{r}}}(d ) = {P_{\mathrm{t}}}{G_{\mathrm{t}}}{G_{\mathrm{r}}}{\left( {\frac{\lambda }{{4\pi }}} \right)^{2}}{d^{- n}}, $$
(1)

where Pr denotes the received signal power and λ denotes the wavelength. The mmWave transmission experiences severe pathloss because of its shorter wavelength. Moreover, the short wavelength of mmWave leads to a lower drain efficiency of the power amplifiers, and the power consumption in baseband (BB) processing is also higher than that in the microwave band [4].

In contrast to conventional MIMO structures whose signal processing is finished digitally at BB, the high cost and power consumption make fully digital processing in the mmWave band impractical [5]. A hybrid analog-digital structure is a feasible approach to be adopted as the mmWave MIMO structure. In the hybrid analog-digital structure, the conventional fully digital processor is replaced by a reduced dimension digital processor and a large-scale analog processor that is composed of phase shifters. However, the analog processor introduces new problems. Both the magnitude and phase of the signal can be adjusted by the fully digital processor, whereas only the phase of the signal can be controlled by the analog processor. This limitation makes the precoding of mmWave MIMO systems difficult. Many precoding schemes have been proposed based on such a hybrid structure. In [6, 7], beam-codebook-based beam steering schemes are proposed to achieve the beamforming of the analog precoders. An orthogonal matching pursuit (OMP)-based spatially sparse precoding scheme is proposed in [5]. By utilizing the scattering nature of the mmWave channels, the precoding algorithm achieves accurate approximation performance of the fully digital structure through an iterative process. Then, the spatially sparse precoding algorithm is extended to the amplify-and-forward (AF) relaying system, which exhibits better performance than the conventional beam steering method [8]. An iterative hybrid precoding design is proposed in [9] to minimize the weighted sum of squared residuals between the optimal full-BB design and the hybrid design. A hybrid precoding design with limited channel state information is also considered in [10]. In [11,12,13], hybrid precoding algorithms are proposed for multiuser mmWave MIMO systems, showing that the sub-connected structure can achieve a higher energy efficiency than the fully-connected structure.

Full-duplex (FD) communication, which refers to transmitting and receiving signals in the same band simultaneously, is also a promising technology for improving the spectral efficiency compared with traditional time division duplex (TDD) or frequency division duplex (FDD) transmission. However, FD transmission can cause severe self-interference (SI) from the transmitting node to the receiving node. Although the state-of-the-art in SI cancellation has made great progress, residual SI still exists [14]. Therefore, residual SI cancellation is necessary in FD systems. FD relaying is a prospective application in the cellular system, which can strengthen communication links, improve system throughput and extend coverage [15]. In addition, FD relaying can reduce time delay and solve the problem of hidden terminals.

According to the work in [16,17,18], FD can be realized in the mmWave band. In [16], a 60 GHz CMOS FD transceiver with a polarization-based antenna is achieved, with a total SI suppression of > 70 dB. The SI in the 60 GHz band is measured and analyzed in [17]. The results show that narrow-beam antennas can effectively suppress the direct SI. Increased antenna spacing and polarization isolation can contribute to SI cancellation [18]. In addition, although the direct SI can be eliminated through beamforming, the reflected SI channel cannot be ignored. Based on the measurement results, we propose an SI model that consists of LOS SI and NLOS SI. The combination of mmWave and FD technologies can achieve notable spectral efficiency and throughput. Meanwhile, the FD relay expands the coverage of the mmWave.

Few studies have been conducted on FD mmWave relaying. In [19, 20], the energy efficiency of FD mmWave relaying is presented. Both mmWave and FD are power-consuming technologies, and the SI cancellation method needs to be selected according to the system design. An OMP-based FD mmWave precoding design is proposed in [21], but the SI is neglected. The degree of freedom is analyzed in the mmWave FD multiuser system with partial channel state information at the transmitter [22]. The uplink channel and downlink channel are assumed to be L-path poor scattering channels, but the SI is still not taken into account. In [23], the SI is considered in an FD mmWave base station. However, the SI is only represented by an attenuation factor. In [24], a line-of-sight (LOS) SI model is applied, and beam-steering-based SI cancellation and constant-amplitude-constrained SI cancellation beamforming schemes are proposed.

In this paper, we consider an FD mmWave relay backhaul system with SI. The combined LOS SI and non-line-of-sight (NLOS) SI FD mmWave SI channel model is proposed first. The LOS SI is the direct link from the transmit antenna to the receive antenna. The NLOS SI is the reflected SI, which is different from microwave band systems. Then, a low-complexity decoupled analog-digital SI cancellation precoding algorithm and an enhanced SI cancellation precoding algorithm are proposed to eliminate the SI and increase the spectral efficiency of the system. Combined with the zero space SI cancellation scheme, the decoupled analog-digital algorithm can significantly eliminate the SI. Compared with half-duplex (HD) transmission, the FD mode can achieve higher spectral efficiency. The enhanced algorithm can achieve higher spectral efficiency, while the decoupled analog-digital algorithm can eliminate the SI with lower computational complexity. The spectral efficiencies of the fully-connected system and the sub-connected system are analyzed. The sub-connected structure reduces the configuration complexity of the hardware components, which also leads to lower spectral efficiency.

The remainder of this paper is organized as follows. In Section 2, we present the system model of the FD mmWave relay backhaul system. The clustered mmWave channel and direct link combined FD mmWave SI channel model is proposed. Then, two SI cancellation precoding algorithms are proposed in Section 3. The precoding algorithm for the sub-connected structure is also proposed. In Section 4, the numerical results of the SI cancellation algorithms are presented. Finally, conclusions are drawn in Section 5.

Notation

In the following, bold lowercase and uppercase letters denote vectors and matrices, respectively. AH denotes the conjugate transpose of A. \({\mathrm {E}}\left [ \cdot \right ]\), \({\text {diag}}\left (\cdot \right )\), ∘, \(\left | {\; \cdot \;} \right |\) and \({\left \| {\: \cdot \:} \right \|_{F}}\) denote the expectation, diagonal matrix, Hadamard product, determinant, and F-norm, respectively.

2 System model

In this section, we present the system model of the relay backhaul shown in Fig. 1. In addition, the combined LOS and NLOS FD mmWave SI channel model is proposed.

Fig. 1
figure 1

System model

The system is composed of three nodes: a source, an FD mmWave relay and a destination. The source is the macrocell base station, and the destination can be a small cell base station. There is no direct link between the source and destination due to the high path loss of mmWave. As shown in Fig. 1, each node adopts a hybrid analog and digital precoding structure. The source node is equipped with NT transmit antennas with NRFS RF chains, and the number of data streams is NS. To realize multistream transmission, the number of RF chains and antennas satisfies NSNRFSNT. The numbers of antennas, RF chains and data streams of the relay and destination are defined in the same way. It is assumed that each MIMO antenna system adopts the hybrid analog-digital architecture shown in Fig. 1. In contrast to the conventional fully digital structure, RF chains are configured between the digital processor and antennas. In this system, each RF chain is connected to all antennas via phase shifters, which is regarded as a fully-connected network.

Then, the NT × 1 transmitted signal of the source can be expressed as

$$ {{\textbf{x}}_{\mathrm{S}}} = {{\textbf{W}}_{{\text{RF}}}}{{\textbf{W}}_{{\text{BB}}}}{{\textbf{s}}_{\mathrm{S}}}, $$
(2)

where WRF is the NT × NRFS RF precoding matrix, WBB is the NRFS × NS BB precoding matrix, and sS is the NS × 1 transmitted data, which satisfies \({\text {E[}}{{\textbf {s}}_{\mathrm {S}}}{\textbf {s}}_{\mathrm {S}}^{\mathrm {H}}{\mathrm {] = }}{{\textbf {I}}_{{N_{\mathrm {S}}}}}\). The power constraint of the source node is \({\left \| {{{\textbf {W}}_{{\text {RF}}}}{{\textbf {W}}_{{\text {BB}}}}} \right \|_{\mathrm {F}}^{2} = {N_{\mathrm {S}}}}\). The signal is transmitted to the relay, and the received signal at the relay is

$$ {{\textbf{y}}_{\mathrm{R}}} = {{\textbf{H}}_{{\text{SR}}}}{{\textbf{x}}_{\mathrm{S}}} + {{\textbf{H}}_{{\text{SI}}}}{{\textbf{x}}_{\mathrm{R}}} + {{\textbf{n}}_{\mathrm{R}}}, $$
(3)

where HSR denotes the nR × NT channel matrix from the source to the relay, HSI denotes the SI channel of the relay, xR denotes the nS × 1 transmitted signal of the relay, and nR denotes the nR × 1 noise vector with i.d.d. \(\mathcal {CN}(0,\sigma _{\mathrm {R}}^{\mathrm {2}})\) at the relay. The first term is the desired signal from the source, and the second term is the SI from the transmit node of the relay. After receiving the signal from the source, the transmitted data of the relay can be expressed as

$$ {{\textbf{s}}_{\mathrm{R}}} = {\textbf{F}}_{{\text{BBR}}}^{\mathrm{H}}{\textbf{F}}_{{\text{RFR}}}^{\mathrm{H}}{{\textbf{y}}_{\mathrm{R}}}, $$
(4)

where FRFR and FBBR denote the nR × NRFR and NRFR × nS RF and BB precoding matrices of the relay receive node, respectively. The transmitted signal at the relay is

$$ {{\textbf{x}}_{\mathrm{R}}} = {{\textbf{F}}_{{\text{RFT}}}}{{\textbf{F}}_{{\text{BBT}}}}{{\textbf{s}}_{\mathrm{R}}}, $$
(5)

where FRFT and FBBT denote the nT × NRFT and NRFT × nS RF and BB precoding matrices of the relay transmit node, respectively. The transmit power constraint of the relay node is \({\left \| {{{\textbf {F}}_{\mathrm {T}}}{{\left ({{{\mathrm {I}}_{{n_{\mathrm {S}}}}} \!\,-\, {\textbf {F}}_{\mathrm {R}}^{\mathrm {H}}{{\textbf {H}}_{{\text {SI}}}}{{\textbf {F}}_{\mathrm {T}}}} \right )}^{- 1}}{\textbf {F}}_{\mathrm {R}}^{\mathrm {H}}({{\textbf {H}}_{{\text {SR}}}}{{\textbf {x}}_{\mathrm {S}}} \,+\, {{\textbf {n}}_{\mathrm {R}}})} \right \|_{\mathrm {F}}^{2} \,=\, {n_{\mathrm {S}}}}\). Substituting Eqs. 3 and 5 into Eq. 4, we can obtain the transmitted data of the relay as

$$ {{\textbf{s}}_{\mathrm{R}}} = {({\textbf{I}} - {\textbf{F}}_{{\text{BBR}}}^{\mathrm{H}}{\textbf{F}}_{{\text{RFR}}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SI}}}}{{\textbf{F}}_{{\text{RFT}}}}{{\textbf{F}}_{{\text{BBT}}}})^{- 1}}{\textbf{F}}_{{\text{BBR}}}^{\mathrm{H}}{\textbf{F}}_{{\text{RFR}}}^{\mathrm{H}}({{\textbf{H}}_{{\text{SR}}}}{{\textbf{x}}_{\mathrm{S}}} + {{\textbf{n}}_{\mathrm{R}}}), $$
(6)

The received signal at the destination can be written as

$$ {{\textbf{y}}_{\mathrm{D}}} = {{\textbf{H}}_{{\text{RD}}}}{{\textbf{x}}_{\mathrm{R}}} + {{\textbf{n}}_{\mathrm{D}}}, $$
(7)

where HRD denotes the NR × nT channel matrix from the relay to the destination and nD denotes the NR × 1 noise vector with i.d.d. \(\mathcal {CN}(0,\sigma _{\mathrm {D}}^{\mathrm {2}})\) at the destination. Finally, the received data of the destination can be expressed as

$$ {{\textbf{s}}_{\mathrm{D}}} = {\textbf{G}}_{{\text{BB}}}^{\mathrm{H}}{\textbf{G}}_{{\text{RF}}}^{\mathrm{H}}{{\textbf{y}}_{\mathrm{D}}} = {\textbf{G}}_{{\text{BB}}}^{\mathrm{H}}{\textbf{G}}_{{\text{RF}}}^{\mathrm{H}}{{\textbf{H}}_{{\text{RD}}}}{{\textbf{x}}_{\mathrm{R}}} + {\textbf{G}}_{{\text{BB}}}^{\mathrm{H}}{\textbf{G}}_{{\text{RF}}}^{\mathrm{H}}{{\textbf{n}}_{\mathrm{D}}}, $$
(8)

where GRF and GBB denote the NR × NRFD and NRFD × nD RF and BB precoding matrices of the destination, respectively. The spectral efficiency of the system is given by

$$ \begin{array}{l} R = {\log_{2}}\left| {{{\textbf{I}}_{{N_{\mathrm{S}}}}} + \frac{1}{{{N_{\mathrm{S}}}}}{\textbf{R}}_{n}^{- 1}\left[ {{{\textbf{G}}^{\mathrm{H}}}{{\textbf{H}}_{{\text{RD}}}}{{\textbf{F}}_{\mathrm{T}}}{{\left( {{{\mathrm{I}}_{{n_{\mathrm{S}}}}} - {\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SI}}}}{{\textbf{F}}_{\mathrm{T}}}} \right)}^{- 1}}{\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SR}}}}{\textbf{W}}} \right]} \right. \\ \left. {{{\left[ {{{\textbf{G}}^{\mathrm{H}}}{{\textbf{H}}_{{\text{RD}}}}{{\textbf{F}}_{\mathrm{T}}}{{\left( {{{\mathrm{I}}_{{n_{\mathrm{S}}}}} - {\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SI}}}}{{\textbf{F}}_{\mathrm{T}}}} \right)}^{- 1}}{\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SR}}}}{\textbf{W}}} \right]}^{\mathrm{H}}}} \right| \end{array} $$
(9)

where W = WRFWBB, FT = FRFTFBBT, FR = FRFRFBBR, G = GRFGBB, and the effective noise of the system is written as

$$\begin{array}{l} {{\textbf{R}}_{n}} \,=\, {\sigma_{n}^{2}}\left[ {\left( {{{\textbf{G}}^{\mathrm{H}}}{{\textbf{H}}_{{\text{RD}}}}{{\textbf{F}}_{\mathrm{T}}}{{\left( {{{\mathrm{I}}_{{n_{\mathrm{S}}}}} \,-\, {\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SI}}}}{{\textbf{F}}_{\mathrm{T}}}} \right)}^{- 1}}{\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}} \right)} \right. \\ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\left. {{{\left( {{{\textbf{G}}^{\mathrm{H}}}{{\textbf{H}}_{{\text{RD}}}}{{\textbf{F}}_{\mathrm{T}}}{{\left( {{{\mathrm{I}}_{{n_{\mathrm{S}}}}} \,-\, {\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SI}}}}{{\textbf{F}}_{\mathrm{T}}}} \right)}^{- 1}}{\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}} \right)}^{\mathrm{H}}} \,+\, {{\textbf{G}}^{\mathrm{H}}}{\textbf{G}}} \right]. \end{array} $$

For HD transmission, the spectral efficiency of the system is expressed as

$$\begin{array}{@{}rcl@{}} R &=& \frac{1}{2}{\log_{2}}\left|\vphantom{\frac{1}{{{N_{\mathrm{S}}}}}} {{\textbf{I}}_{{N_{\mathrm{S}}}}}\right.\\ &&\left.+ \frac{1}{{{N_{\mathrm{S}}}}}{\textbf{R}}_{n}^{- 1}\left[ {{{\textbf{G}}^{\mathrm{H}}}{{\textbf{H}}_{{\text{RD}}}}{{\textbf{F}}_{\mathrm{T}}}{\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SR}}}}{\textbf{W}}} \right] \right.\!\!\left. {{{\left[ {{{\textbf{G}}^{\mathrm{H}}}{{\textbf{H}}_{{\text{RD}}}}{{\textbf{F}}_{\mathrm{T}}}{\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SR}}}}{\textbf{W}}} \right]}^{\mathrm{H}}}} \right|,\\ \end{array} $$
(10)

where effective noise is \({{\textbf {R}}_{n}} \,=\, {\sigma _{n}^{2}}\left [\vphantom {\left (F_{1}^{2}\right )^{2}} \left ({{{\textbf {G}}^{\mathrm {H}}}{{\textbf {H}}_{{\text {RD}}}}{{\textbf {F}}_{\mathrm {T}}}{\textbf {F}}_{\mathrm {R}}^{\mathrm {H}}} \right )\right .\) \(\left .{{{\left ({{{\textbf {G}}^{\mathrm {H}}}{{\textbf {H}}_{{\text {RD}}}}{{\textbf {F}}_{\mathrm {T}}}{\textbf {F}}_{\mathrm {R}}^{\mathrm {H}}} \right )}^{\mathrm {H}}} \,+\, {{\textbf {G}}^{\mathrm {H}}}{\textbf {G}}} \right ]\), the SI of the set to 0 and \(\frac {1}{2}\) denotes the orthogonal time transmission loss.

2.1 mmWave channel model

The short wavelength makes mmWave different from a traditional MIMO system. As illustrated in [3], its short wavelength leads to a significantly high path loss. The short wavelength also enables large numbers of antennas to be packed into a small volume, which causes higher antenna correlation and creates sparse scattering channel environments. Based on the Saleh-Valenzuela model [25], the clustered mmWave channel is described as the sum of Nclu scattering clusters, each of which consists of Npath propagation paths [5, 26, 27]. The channel matrix H can be written as

$$ {\textbf{H}} = \sqrt {\frac{{{N_{\mathrm{t}}}{N_{\mathrm{r}}}}}{{{N_{{\text{clu}}}}{N_{{\text{path}}}}}}} \sum\limits_{i = 1}^{{N_{{\text{clu}}}}} {\sum\limits_{l = 1}^{{N_{{\text{path}}}}} {{\alpha_{il}}{{\Psi}_{\mathrm{r}}}(\phi_{il}^{\mathrm{r}}){{\Psi}_{t}}{{(\phi_{il}^{\mathrm{t}})}^{H}}} } , $$
(11)

where Nt and Nr are the numbers of transmitting and receiving antennas of the corresponding channel and αil is the complex gain of the l-th path in the i-th scattering cluster. The functions \({{{\Psi }_{\mathrm {r}}}(\phi _{il}^{\mathrm {r}})}\) and \({{{\Psi }_{t}}(\phi _{il}^{\mathrm {t}})}\) represent the normalized receive and transmit array response vectors. The angles \({\phi _{il}^{\mathrm {r}}}\) and \({\phi _{il}^{\mathrm {t}}}\) are the azimuth angles of departure and arrival, respectively. αil is the complex Rayleigh channel coefficient, which satisfies \(\mathcal {CN}(0,\sigma _{\alpha i}^{\mathrm {2}})\), where \(\sigma _{\alpha i}^{\mathrm {2}}\) denotes the average power of the i-th scattering cluster. The array response function is determined by the structure of the antenna array configuration. For a uniform linear array (ULA), the function can be expressed as

$$ {{\Psi}_{{\text{ULA}}}}(\phi ) = \frac{1}{{\sqrt N }}{\left[ {1,\;{e^{j\frac{{2\pi }}{\lambda }d\sin (\phi )}},\;...,{e^{j\frac{{2\pi }}{\lambda }(N - 1)d\sin (\phi )}}} \right]^{T}}, $$
(12)

where d is the interelement spacing and λ is the carrier wavelength. The angles are randomly distributed and satisfy a Laplacian distribution, which is expressed as

$$ {p_{\phi} }(\phi ) = \left\{ {\begin{array}{*{20}{c}} {\frac{\beta }{{\sqrt 2 {\sigma_{\phi} }}}{e^{- \left| {\frac{{\sqrt 2 \phi }}{{{\sigma_{\phi} }}}} \right|}}} & {{\text{if}}\phi \in [ - \pi ,\pi )} \\ 0 & {{\text{otherwise}}} , \end{array}} \right. $$
(13)

where \(\beta = 1/(1 - {e^{- \sqrt 2 \pi /{\sigma _{\phi } }}})\) and σϕ is the standard deviation of the power azimuth spectrum. The sequential summation in Eq. 11 can be rewritten in matrix form as

$$ {\mathbf{H}} = \sum\limits_{i = 1}^{{N_{{\text{clu}}}}} {{\mathbf{\mathrm{A}}}_{i}^{r}{{\mathbf{{\Lambda} }}_{i}}({\mathbf{\mathrm{A}}}_{i}^{t}} {)^{H}} = {{\mathbf{\mathrm{A}}}^{r}}{\mathbf{{\Lambda} }}{{\mathbf{\mathrm{A}}}^{t}}, $$
(14)

where \({\mathbf {\mathrm {A}}}_{i}^{r} = [\begin {array}{*{20}{c}} {{\Psi }_{i1}^{r}} & {{\Psi }_{i2}^{r}} & {...} & {{\Psi }_{i{N_{{\text {path}}}}}^{r}} \end {array}]\), \({\mathbf {\mathrm {A}}}_{i}^{t} = [\begin {array}{*{20}{c}} {{\Psi }_{i1}^{t}} & {{\Psi }_{i2}^{t}} & {...}\end {array}\) \(\begin {array}{*{20}{c}} {{\Psi }_{i{N_{{\text {path}}}}}^{t}} \end {array}]\), and the coefficient matrix \({{\mathbf {{\Lambda } }}_{i}} = diag[\begin {array}{*{20}{c}} {\gamma {\alpha _{i1}}}\end {array}\) \(\begin {array}{*{20}{c}} {\gamma {\alpha _{i2}}} & {...} & {\gamma {\alpha _{i{N_{{\text {path}}}}}}} \end {array}]\). The matrices Ar, Λ and At are composed of \({\mathbf {\mathrm {A}}}_{i}^{r}\), Λi and \({\mathbf {\mathrm {A}}}_{i}^{t}\) in the same way.

2.2 mmWave SI channel model

In contrast to conventional microwave systems, FD mmWave transmission can receive NLOS SI because of its high beamforming gain. In addition, the LOS SI also needs to be considered. Therefore, the SI of the FD mmWave relay is composed of two parts: LOS SI and NLOS SI. The NLOS SI results from reflection from nearby obstacles. We assume the same clustered mmWave channel with an intensity coefficient of ηNLOS. The LOS SI is the direct link from the transmit antennas to the receive antennas, which can be modeled as a near-field model. We adopt the transmit and receive antenna configuration shown in Fig. 2. The angle between the nR × 1 ULA and nT × 1 ULA is ψ. The distance between the m th antenna of nR and the n th antenna of nT is dmn, and the corresponding included angles are 𝜃mn and φmn. The distance between the adjacent antennas within a ULA is \(d = \frac {\lambda }{2}\), and the initial distances are a0 and b0. The coefficient of the channel element is

$$ h_{mn}^{{\text{SI}}} = \alpha {e^{- j\frac{{2\pi }}{\lambda }{d_{mn}}}}, $$
(15)

where α is the normalization coefficient and the distance dmn is expressed as

$$ {d_{mn}} \,=\, \sqrt {{{[{a_{0}} \,+\, (m \,-\, 1)d]}^{2}} \,+\, {{[{b_{0}} \,+\, (n \,-\, 1)d]}^{2}} \,-\, 2[{a_{0}} \,+\, (m \,-\, 1)d][{b_{0}} \,+\, (n \,-\, 1)d]\cos \psi }. $$
(16)

The SI channel matrix is written as

$$ {{\mathbf{H}}_{{\text{LOS}}}} = [{{\mathbf{\alpha }}^{{\text{SI}}r}}{({{\mathbf{\alpha }}^{{\text{SI}}t}})^{H}}] \circ {{\mathbf{H}}^{{\text{SI}}}}, $$
(17)

where the antenna array response vectors of the receive and transmit arrays are αSIr = \([1e^{j\frac {{2\pi d}}{\lambda } \sin {\phi _{r}}}...\;{e^{j\frac {{2\pi d}}{\lambda } ({n_{R}} - 1)\sin {\phi _{r}}}}{{]}^{H}}\) and \({\alpha ^{{\text {SI}}t}}={[1\;{e^{j\frac {{2\pi d}}{\lambda }\sin {\phi _{t}}}}\;...\;{e^{j\frac {{2\pi d}}{\lambda }({n_{T}} - 1)\sin {\phi _{t}}}}]^{H}}\), and HSI \(= \{ h_{mn}^{{\text {SI}}}|n = 1,...,{n_{T}},m = 1,...,{n_{R}}\} \). The operator ∘ denotes the Hadamard product, which takes two matrices of the same dimensions and multiplies the corresponding elements of the two matrices. Finally, the FD mmWave SI channel can be expressed as

$$ {{\textbf{H}}_{{\text{SI}}}} = {\eta_{{\text{LOS}}}}{{\textbf{H}}_{{\text{LOS}}}} + {\eta_{{\text{NLOS}}}}{{\textbf{H}}_{{\text{NLOS}}}}, $$
(18)

where ηLOS denotes the intensity coefficient of the LOS channel portion.

Fig. 2
figure 2

FD LOS SI model

3 FD mmWave relay precoding design

In this section, to realize low complexity and a low processing delay, we propose a decoupled analog-digital SI cancellation precoding algorithm. In addition, the MIMO architectures of fully-connected and sub-connected mmWave structures are analyzed.

The aim of the precoding design is to maximize the spectral efficiency in Eq. 9 by designing all the matrices WBB, WRF, FBBT, FRFT, FBBR, FRFR, GBB and GRF. Combining with the constraint conditions, the problem can be expressed as

$$ \begin{array}{*{20}{c}} {\mathop {\max }\limits_{{\textbf{W}},{{\textbf{F}}_{\mathrm{R}}},{{\textbf{F}}_{\mathrm{T}}},{\textbf{G}}} } & \begin{array}{l} \!\!R \,=\, {\log_{2}}\left| {{{\textbf{I}}_{{N_{\mathrm{S}}}}} \!\,+\, \frac{1}{{{N_{\mathrm{S}}}}}{\textbf{R}}_{n}^{- 1}\left[ {{{\textbf{G}}^{\mathrm{H}}}{{\textbf{H}}_{{\text{RD}}}}{{\textbf{F}}_{\mathrm{T}}}{{\left( {{{\mathrm{I}}_{{n_{\mathrm{S}}}}} \!\!- \! {\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SI}}}}{{\textbf{F}}_{\mathrm{T}}}} \right)}^{- 1}}{\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SR}}}}{\textbf{W}}} \right]} \right. \\ \left. \;\;\;\;\;\;\;{{{\left[ {{{\textbf{G}}^{\mathrm{H}}}{{\textbf{H}}_{{\text{RD}}}}{{\textbf{F}}_{\mathrm{T}}}{{\left( {{{\mathrm{I}}_{{n_{\mathrm{S}}}}} - {\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SI}}}}{{\textbf{F}}_{\mathrm{T}}}} \right)}^{- 1}}{\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SR}}}}{\textbf{W}}} \right]}^{\mathrm{H}}}} \right| \end{array} \\ {{\mathrm{s}}{\mathrm{.t}}{\mathrm{.}}} & \begin{array}{l} \left\| {{{\textbf{W}}_{{\text{RF}}}}{{\textbf{W}}_{{\text{BB}}}}} \right\|_{\mathrm{F}}^{2} = {N_{\mathrm{S}}} \\ \left\| {{{\textbf{F}}_{\mathrm{T}}}{{\left( {{{\mathrm{I}}_{{n_{\mathrm{S}}}}} - {\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SI}}}}{{\textbf{F}}_{\mathrm{T}}}} \right)}^{- 1}}{\textbf{F}}_{\mathrm{R}}^{\mathrm{H}}({{\textbf{H}}_{{\text{SR}}}}{{\textbf{x}}_{\mathrm{S}}} + {{\textbf{n}}_{\mathrm{R}}})} \right\|_{\mathrm{F}}^{2} = {n_{\mathrm{S}}}\; \\ {{\textbf{W}}_{{\text{RF}}}} \in {{\mathcal{W}}_{{\text{RF}}}},{{\textbf{F}}_{{\text{RFT}}}} \in {{\mathcal{F}}_{{\text{RFT}}}},{{\textbf{F}}_{{\text{RFR}}}} \in {{\mathcal{F}}_{{\text{RFR}}}},{{\textbf{G}}_{{\text{RF}}}} \in {{\mathcal{G}}_{{\text{RF}}}}, \end{array} \end{array} $$
(19)

where \(\left \| {{{\textbf {F}}_{\mathrm {T}}}{{\left ({{{\mathrm {I}}_{{n_{\mathrm {S}}}}} - {\textbf {F}}_{\mathrm {R}}^{\mathrm {H}}{{\textbf {H}}_{{\text {SI}}}}{{\textbf {F}}_{\mathrm {T}}}} \right )}^{- 1}}{\textbf {F}}_{\mathrm {R}}^{\mathrm {H}}({{\textbf {H}}_{{\text {SR}}}}{{\textbf {x}}_{\mathrm {S}}} + {{\textbf {n}}_{\mathrm {R}}})} \right \|_{\mathrm {F}}^{2} = {n_{\mathrm {S}}}\) denotes the power constraint at the relay and \(\left \| {{{\textbf {W}}_{{\text {RF}}}}{{\textbf {W}}_{{\text {BB}}}}} \right \|_{\mathrm {F}}^{2} = {N_{\mathrm {S}}}\) denotes the power constraint at the source. \({\mathcal {W}}_{{\text {RF}}}\), \({\mathcal {F}}_{{\text {RFT}}}\), \({\mathcal {F}}_{{\text {RFR}}}\), and \({\mathcal {G}}_{{\text {RF}}}\) are the feasible sets of the RF precoders, which denote the sets of matrices with constant-magnitude elements. The constraint results from the limitation of the phase shifters, which can only adjust the phase of each coefficient as \({\left \{ {{{\textbf {W}}_{{\text {RF}}}}} \right \}_{ij}} = {e^{\theta ij}}\), \(\forall i \in \left \{ {1,...,{N_{\mathrm {T}}}} \right \}\), \(\forall j \in \left \{ {1,...,{N_{{\text {RFS}}}}} \right \}\).

3.1 Decoupled analog-digital SI cancellation precoding design

The specific structure of mmWave that leads to the nonconvex limitation of RF precoders makes the optimization problem intractable. We propose a decoupled analog-digital SI cancellation precoding algorithm in which the RF precoders and BB precoders are designed separately. The precoding design is divided into two parts: analog precoding and digital precoding. The entries of RF precoders can only change their phases with constant magnitude because of the structure of the phase shifters. Therefore, the ability of RF precoders is limited.

The RF precoders are designed to enhance the desired channel in the channel matrix by choosing the phases of the conjugate of the channel. Then, the interference is eliminated through BB precoding design, and the SI is considered at the same time. The precoding designs of WBB, WRF, FBBR and FRFR are on the basis of HSR. The channel state information HSR is an nR × NT matrix, and the RF precoder of the source WRF is an NT × NRFS matrix. We choose the phases of the first NRFS row vectors of HSR as the RF precoder. The precoding matrix WRF can be expressed as

$$ {{\textbf{W}}_{{\text{RF}}}}{\mathrm{ = }}\left[ {\begin{array}{*{20}{c}} {{e^{j(- \arg (h_{{\mathrm{11}}}^{{\text{SR}}}))}}} & {{e^{j(- \arg (h_{{\mathrm{21}}}^{{\text{SR}}}))}}} & {\cdots} & {{e^{j(- \arg (h_{{N_{{\text{RFS}}}}{\mathrm{1}}}^{{\text{SR}}}))}}} \\ {{e^{j(- \arg (h_{{\mathrm{12}}}^{{\text{SR}}}))}}} & {{e^{j(- \arg (h_{{\mathrm{22}}}^{{\text{SR}}}))}}} & {\cdots} & {{e^{j(- \arg (h_{{N_{{\text{RFS}}}}{\mathrm{2}}}^{{\text{SR}}}))}}} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {{e^{j(- \arg (h_{{\mathrm{1}}{N_{\mathrm{T}}}}^{{\text{SR}}}))}}} & {{e^{j(- \arg (h_{{\mathrm{2}}{N_{\mathrm{T}}}}^{{\text{SR}}}))}}} & {\cdots} & {{e^{j(- \arg (h_{{N_{{\text{RFS}}}}{N_{\mathrm{T}}}}^{{\text{SR}}}))}}} \end{array}} \right], $$
(20)

where \({h_{ij}^{{\text {SR}}}}\) denotes the i th row and the j th column of the channel matrix HSR and \({\arg (h_{ij}^{{\text {SR}}})}\) denotes the phase of the corresponding entry. Then, we obtain the nR × NRFS partly equivalent channel HSReqt = HSRWRF. The RF precoder FRFR can be obtained by utilizing the phases of HSReqt without changing the position of the entries as

$$ {{\textbf{F}}_{{\text{RFR}}}}{\mathrm{ = }}\left[ {\begin{array}{*{20}{c}} {{e^{j\arg (h_{11}^{{\text{SReqt}}})}}} & {{e^{j\arg (h_{12}^{{\text{SReqt}}})}}} & {\cdots} & {{e^{j\arg (h_{1{N_{{\text{RFR}}}}}^{{\text{SReqt}}})}}} \\ {{e^{j\arg (h_{21}^{{\text{SReqt}}})}}} & {{e^{j\arg (h_{22}^{{\text{SReqt}}})}}} & {\cdots} & {{e^{j\arg (h_{2{N_{{\text{RFR}}}}}^{{\text{SReqt}}})}}} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {{e^{j\arg (h_{{n_{R}}1}^{{\text{SReqt}}})}}} & {{e^{j\arg (h_{{n_{R}}2}^{{\text{SReqt}}})}}} & {\cdots} & {{e^{j\arg (h_{{n_{R}}{N_{{\text{RFR}}}}}^{{\text{SReqt}}})}}} \end{array}} \right], $$
(21)

where \({h_{mn}^{{\text {SReqt}}}}\) denotes the entries of the partly equivalent channel. After designing the RF precoders, we can obtain the NRFR × NRFS equivalent channel \({{\textbf {H}}_{{\text {SReq}}}} = {\textbf {F}}_{{\text {RFR}}}^{\mathrm {H}}{{\textbf {H}}_{{\text {SR}}}}{{\textbf {W}}_{{\text {RF}}}}\). The BB precoders are designed based on the equivalent channel through singular value decomposition (SVD). The SVD of the equivalent channel is written as HSReq = USReqΣSReqVSReq, and the BB precoders WBB and FBBR can be obtained by taking the corresponding column vectors as

$$ \begin{array}{l} {{\textbf{W}}_{{\text{BB}}}} = {{\textbf{V}}_{{\text{SReq}}}}(:,[1:{N_{\mathrm{S}}}]) \\ {{\textbf{F}}_{{\text{BBR}}}} = {{\textbf{U}}_{{\text{SReq}}}}(:,[1:{n_{\mathrm{S}}}]) . \end{array} $$
(22)

Similarly, the RF and BB precoders FBBT, FRFT, GBB and GRF can be derived by following the above steps with the channel matrix HRD.

However, the SI is not eliminated. As illustrated in Eq. 9, the term that relates to SI is written as \(\left ({{\mathrm {I}}_{{n_{\mathrm {T}}}}} - {\textbf {F}}_{{\text {BBR}}}^{\mathrm {H}}\right .\) \(\left .{\textbf {F}}_{{\text {RFR}}}^{\mathrm {H}}{{\textbf {H}}_{{\text {SI}}}}{{\textbf {F}}_{{\text {RFT}}}}{{\textbf {F}}_{{\text {BBT}}}} \right )^{- 1}\). The aim of SI cancellation is to design the precoding matrices and eliminate the SI such that \({\textbf {F}}_{{\text {BBR}}}^{\mathrm {H}}{\textbf {F}}_{{\text {RFR}}}^{\mathrm {H}}{{\textbf {H}}_{{\text {SI}}}}{{\textbf {F}}_{{\text {RFT}}}}{{\textbf {F}}_{{\text {BBT}}}}{\mathrm { = }}{\textbf {0}}\). To eliminate the interference, the BB precoder FBBT needs to be redesigned to utilize the zero space of the SI channel. The analog equivalent channel HSIeq can be expressed as

$$ {{\textbf{H}}_{{\text{SIeq}}}} = {\textbf{F}}_{{\text{RFR}}}^{\mathrm{H}}{{\textbf{H}}_{{\text{SI}}}}{{\textbf{F}}_{{\text{RFT}}}}. $$
(23)

The SVD of the equivalent channel can be expressed as

$$ {{\textbf{H}}_{{\text{SIeq}}}} = {{\textbf{U}}_{{\text{SIeq}}}}{{\mathbf{{\Sigma} }}_{{\text{SIeq}}}}{{\textbf{V}}_{{\text{SIeq}}}} = {{\textbf{U}}_{{\text{SIeq}}}}{{\mathbf{{\Sigma} }}_{{\text{SIeq}}}}[{\textbf{V}}_{{\text{SIeq}}}^{1}{\textbf{V}}_{{\text{SIeq}}}^{0}], $$
(24)

where USIeq is the left-singular matrix, ΣSIeq is the singular value matrix, and VSIeq is the right-singular matrix. \({\textbf {V}}_{{\text {SIeq}}}^{1}\) and \({\textbf {V}}_{{\text {SIeq}}}^{0}\) are the right-singular vectors corresponding to nonzero and zero singular values, respectively. In order to ensure effective SI cancellation through utilizing the zero space, the number of RF chains and data streams of the relay has to satisfy the inequality NRFTNRFR + nS, namely, the transmit node needs at least nS more RF chains than the receive node. Then, we have

$$ {{\textbf{H}}_{{\text{SIeq}}}}{\textbf{V}}_{{\text{SIeq}}}^{0} = {\textbf{0}}. $$
(25)

Additionally, FBBT can be decided by choosing the column vectors of \({\textbf {V}}_{{\text {SIeq}}}^{0}\) that satisfy its dimension requirement. Finally, the power constraint in Eq. 19 needs to be satisfied.

The complete description of the designed algorithm is shown in Algorithm 1. The last two steps, 11 and 12, are the power constraint computation. Note that the precoder of the destination can be obtained by utilizing the same method in steps 1-6.

figure a

3.2 Enhanced decoupled analog-digital SI cancellation algorithm

In section III A, the RF precoders are designed at the transmitting end, and the first NRF row vectors of the channel matrix are chosen. In this section, a joint transmitting and receiving precoding design is proposed to further improve the performance.

First of all, the channel matrix HSR can be denoted in vector form as \({{\textbf {H}}_{{\text {SR}}}}{\mathrm { = }}{\left [ {\begin {array}{*{20}{c}} {{\textbf {h}}_{1}^{{\text {SR}}}} & \!{{\textbf {h}}_{2}^{{\text {SR}}}} &\! {\cdots } & \!{{\textbf {h}}_{{n_{\mathrm {R}}}}^{{\text {SR}}}} \end {array}} \right ]^{T}}\), and the aim of RF precoding design is to choose NRFS vectors from the nR row vectors. The NRFS vectors are chosen by comparing the values of the diagonal elements of the matrix \({{\textbf {H}}_{{\text {SR}}}}{\textbf {H}}_{{\text {SR}}}^{H}\). The largest NRFS diagonal elements and the corresponding vectors are chosen. However, the order of the chosen vectors is not decided by the diagonal elements. It is determined by the singular value of the equivalent channel \({{\textbf {H}}_{{\text {SReq}}}} = {\textbf {F}}_{{\text {RFR}}}^{\mathrm {H}}{{\textbf {H}}_{{\text {SR}}}}{{\textbf {W}}_{{\text {RF}}}}\). The precoding design is illustrated in Algorithm 2. In step 1 and step 2, the NRFS vectors are chosen as a basic set of the RF precoder. The singular values of all the permutations of the set are computed in the loop iteration. Both the transmitting RF precoder and the receiving RF precoder are designed, and the combination with the maximal Σ is the desired precoding matrix. The BB precoding design can be obtained by applying SVD and the zero space scheme.

figure b

3.3 Precoding design for sub-connected structure

The mmWave communication is a power-consuming technology. The short wavelength of mmWave leads to a lower drain efficiency and higher power consumption in circuits. The hybrid analog and digital structure in Fig. 1 is an alternative solution for the expensive and power-consuming fully digital structure. The hybrid structure can be further simplified to a sub-connected structure, which is shown in Fig. 3. The precoding algorithm for the subconnected structure is proposed in this section.

Fig. 3
figure 3

Fully-connected structure and sub-connected structure

As shown in Fig. 3, each RF chain is connected to all the antennas in the fully-connected structure. However, the sub-connected RF chain only connects a small group of the antennas, and each group contains M = NT/NRF antennas. The difference between these two structures can be revealed in the formation of the RF precoding matrices. The RF precoder of the sub-connected structure is written as

$$ {{\textbf{W}}_{{\text{RF}}}}{\mathrm{ = }}\left[ {\begin{array}{*{20}{c}} {{\textbf{w}}_{{\mathrm{RF1}}}^{{\text{sub}}}} & {\mathrm{0}} & {\cdots} & {\mathrm{0}} \\ {\mathrm{0}} & {{\textbf{w}}_{{\mathrm{RF2}}}^{{\text{sub}}}} & {\cdots} & {\mathrm{0}} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {\mathrm{0}} & {\mathrm{0}} & {\cdots} & {{\textbf{w}}_{{\text{RF}}{N_{{\text{RFS}}}}}^{{\text{sub}}}} \end{array}} \right] $$
(26)

where \({\textbf {w}}_{{\text {RF}}i}^{{\text {sub}}}\) denotes the M × 1 precoding vector of the i th RF chain. Compared with the precoding matrix of the fully-connected structure in Eq. 20, most of the entries in Eq. 26 are zeros. The RF precoding designs of the precoding vectors are expressed as

$$ w_{{\text{RF}}ij}^{{\text{sub}}} = {e^{j(- \arg (h_{ji}^{{\text{SR}}}))}},(i \,=\, (j - 1)M + 1, {\ldots} ,jM;j \,=\, 1, {\ldots} ,{N_{{\text{RF}}}}) $$
(27)

where \(w_{{\text {RF}}ij}^{{\text {sub}}}\) denotes entries of the precoding vectors \({{\textbf {w}}_{{\text {RF}}i}^{{\text {sub}}}}\). Therefore, there are only NT nonzero entries in the matrix, which will cause a performance loss compared with the fully-connected structure. However, the simplified structure is equipped with less phase shifters and power amplifiers, which can decrease the hardware design complexity. For both of the structures, the numbers of phase shifters, power amplifiers, splitters and combiners are expressed as

$$\begin{array}{@{}rcl@{}} &&{N_{PA\_fully}} = {N_{T}}({N_{RF}} + 1)\;\;\;\;\;{N_{PA\_sub}} = {N_{T}}\\ &&{N_{PS\_fully}} = {N_{T}}{N_{RF}}\;\;\;\;\;\;\;\;\;\;\;\;\;\;{N_{PS\_sub}} = {N_{T}}\\ &&{N_{S/C\_fully}} = {N_{T}}\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;{N_{S/C\_sub}} = 0 \end{array} $$
(28)

where NT denotes the number of antennas and NRF denotes the number of RF chains. \({N_{PA\_fully}}\) and \({N_{PA\_sub}}\) denote the numbers of power amplifiers of the fully-connected structure and sub-connected structure, respectively. \({N_{PS\_fully}}\) and \({N_{PS\_sub}}\) denote the numbers of the phase shifters of the structures. \({N_{S/C\_fully}}\) and \({N_{S/C\_sub}}\) denote the numbers of splitters or combiners of the two structures. With a large number of antennas, the sub-connected structure can save a great quantity of hardware components.

4 Numerical results

In this section, the numerical results are presented to show the performance of the decoupled analog-digital SI cancellation precoding algorithm and the enhanced decoupled analog-digital precoding algorithm. The influence of different hardware structure parameters is analyzed based on the algorithm. In addition, the spectral efficiencies of the fully-connected and sub-connected structures are discussed.

We assume that the channel has Nclu = 3 scattering clusters with Npath = 10 paths per cluster. And the complex gain αil satisfies \(\mathcal {CN}(0,1)\). A ULA with interelement spacing \(d = \frac {\lambda }{2}\) is assumed. The bandwidth of the system is 2 GHz with a central carrier frequency fc = 60 GHz. The MIMO system configuration of the hybrid analog-digital structure is NT = 32, NRF = 4 and NS = 2. For simplicity, we assume that \(\sigma _{\mathrm {S}}^{2} = \sigma _{\mathrm {R}}^{2} = \sigma _{\mathrm {D}}^{2} = {\sigma ^{2}}\), and the signal-to-noise ratio (SNR) is defined as SNR = NS/σ2.

4.1 Decoupled analog-digital SI cancellation precoding algorithm

Figure 4 presents the results of the decoupled analog-digital SI cancellation precoding algorithm and the enhanced precoding algorithm. The red lines show the performance of HD and FD in ideal conditions, where the SI is assumed to be perfectly eliminated. Compared with the HD communication system, the FD relay can achieve approximately two-times greater spectral efficiency. However, the existence of SI causes a serious performance loss for the FD system, which leads to worse spectral efficiency than the HD system. The blue lines show the performance of the SI cancellation scheme. It can achieve high spectral efficiency with only a slight performance loss. The green line represents the enhanced algorithm, which realizes higher spectral efficiency than the original algorithm. The proposed algorithms can effectively eliminate the SI and help the FD system achieve higher performance than the HD system.

Fig. 4
figure 4

Decoupled analog-digital algorithm and enhanced decoupled analog-digital algorithm

In Figs. 56 and 7, the influence of different structure parameters is considered. In Fig. 5, there are three different configurations, A, B and C, on the number of data streams NS, the number of RF chains NRF and the number of antennas NT. Configuration A is equipped with NT = 32 antennas together with NRF = 8 RF chains and NS = 3 data streams, which is written as (3,8,32). Configurations B and C are (3,8,64) and (3,8,128), respectively. Each configuration contains two conditions: FD in ideal conditions and the SI cancellation scheme. With a large number of antennas, the spectral efficiency is significantly improved. Compared with configuration A, the gap between the two conditions is reduced in configuration C. Figure 6 shows the spectral efficiencies of the relay system with different numbers of RF links. Configurations A, B and C are (3,4,64), (3,8,64) and (3,16,64), respectively. The results show that the number of RF chains has little influence on the spectral efficiency, even if more RF chains can achieve higher performance. In Fig. 7, the spectral efficiencies of different numbers of data streams are presented. The configurations A, B and C are NS = 1, NS = 2 and NS = 4, respectively. With more data streams, both the spectral efficiency and the growth trend increase.

Fig. 5
figure 5

Spectral efficiency comparison with different numbers of antennas

Fig. 6
figure 6

Spectral efficiency comparison with different numbers of RF links

Fig. 7
figure 7

Spectral efficiency comparison with different numbers of data streams

4.2 Spectral efficiency analysis for sub-connected structure

The spectral efficiencies of the fully-connected and sub-connected structures are shown in Fig. 8. Both ideal conditions and SI cancellation performance of the sub-connected structure are presented. The simplified antenna configuration decreases its performance, and it can achieve approximately half the spectral efficiency of the fully-connected structure.

Fig. 8
figure 8

Spectral efficiencies of fully-connected and sub-connected structures

In Fig. 9, three groups of results are presented to show the effects of the parameter NT. Configurations A, B and C are (3,8,16), (3,8,32) and (3,8,64), respectively. The gap within the group increases as the number of antennas doubles, which can be explained by Eqs. 26 and 28. Moreover, the spectral efficiency of the sub-connected structure increases slowly with a large number of antennas compared with the fully-connected structure.

Fig. 9
figure 9

Fully-connected and sub-connected structures with different numbers of antennas

5 Conclusions

In this paper, we propose an LOS SI and NLOS SI combined FD mmWave SI channel model in an FD mmWave relay backhaul system. Then, a decoupled analog-digital SI cancellation precoding algorithm and enhanced SI cancellation algorithm are proposed to eliminate the SI in the system. Compared with HD transmission, the FD mode can achieve higher spectral efficiency. The decoupled analog-digital algorithm can eliminate the SI and achieve near ideal performance. The enhanced precoding algorithm can achieve higher performance by increasing complexity. The sub-connected structure is also analyzed. With a large number of antennas, the sub-connected structure can realize a low hardware complexity design. However, the spectral efficiency of the simplified structure is limited.