1 Introduction

Due to the increase in not only the quantity of users but also the quality of wireless systems, wireless broadband services have driven high transport capacity requirements among cellular networks [1]. As a result, the deployment of wireless infrastructure will get more dense and heterogeneous in the near future [2]. To achieve such higher data rate systems, backhaul as the backbone links between the macro-cell and many small-cells in heterogeneous networks (HetNets) is becoming an emerging technology. In the traditional way, wired backhaul has shown their advantages of high reliability communications, yet deploying the large-scale wired links would lead to an ineffective increase in the costs of maintaining all the connections. For this reason, wireless backhaul is considered as an alternative solution since it offers cost-efficiency and flexibility in the practical systems. In spite of satisfying many requirements for the availability of the backhaul connections, wireless backhaul is not completely reliable compared to wired backhaul due to the existence of non-line-of-sight (n-LOS) propagation and fading of transmission signals. [3, 4].

There are many existing studies that investigated unreliable wireless backhaul links. For example, in [5], the authors analyzed the impact of heterogeneous backhaul on a femtocell network by using game theory. In [6], the impact of unreliable backhaul connections on the performance of Coordinated Multi-Point (CoMP) techniques has been investigated in the cooperative downlink system. For a coordinated multi-point system under Rayleigh fading, the authors in [7] have proposed an analytical framework for the performance analysis with unreliable backhaul links. Taking into account the limited resources such as the number of transmitters, interferers and backhaul reliability, the performance of a cooperative wireless network has been investigated in [8]. It has been proved that in the high transmit power region, the aggregate interference and the unreliable backhaul are responsible for the asymptotic behavior of the system. For most of research works, backhaul reliability is shown as one of the key parameters that have significant impact on the system performance.

Cooperative systems in dense networks aim to extend the coverage or enhance the system capacity [9]. The fundamental idea of the cooperative transmission is that, the desired signal transmitted by a macro-cell to the destination is captured by multiple small-cell transmitters which act as relaying nodes. After receiving the signal in the first stage, those small-cell transmitters process and immediately retransmit the signal to the destination in the second stage. Then the receiver combines all the signals from the macro-cell and small-cell transmitters. Therefore, the diversity gain can be improved by taking advantage of the multiple receptions at various transmitters and transmission paths[10]. Several papers considering aggregation schemes can be found in the literature. For relay selection over Rayleigh fading channels, the authors in [11] investigated the secrecy performance of three different diversity combining schemes, namely maximum ratio combining (MRC), selection combining (SC), switch-and-stay combining (SSC). In [12], the authors analyzed the security of MRC systems with the channel-state-information (CSI) at the eavesdropper being available/not available. For a cyclic-prefix single carrier (CP-SC) system, the best relay selection scheme has been employed to analyze the performance in cognitive radio (CR) sharing spectrum [13].

As the demand for additional bandwidth continues to grow exponentially [14], many experts have sought solutions to efficiently deploy the available licensed spectrum. In recent years, the investigation on CR technologies has attracted the research community as a key factor to improve the spectrum scarcity in HetNets [15, 16]. By allowing the secondary users to share the same spectrum which is originally allocated to primary users, the spectrum efficiency can be significantly enhanced. One of CR applications is cognitive relay networks. Under Nakagama-m fading, the authors in [17] analyzed the performance impacts of amplify-and-forward (AF) protocol subject to the transmit power constraints at the source and relay node. In [18], the authors investigated the transmit antenna selection with receive generalized selection combining (TAS/GSC) in CR networks over Nakagami-m fading. To study the impact of cooperative AF relaying in spectrum sharing system, the partial and opportunistic relay selection strategies have been investigated in [19].

Since the spectrum in primary networks has not been well utilized, it is important to integrate the CR technologies in the dense communication networks. To the best of the authors’ knowledge, most of previous works only considered CR by neglecting the impact of unreliable backhaul [20,21,22]. Therefore, we are motivated to analyze the performance of such systems. With the existence of CR networks in cooperative systems, the spectrum efficiency is then improved effectively. However, due to the nature of sharing spectrum and wireless connections, the emitted interference to the primary network and the unreliability of the backhaul links should be taken into account. Based on those considerations, our main contributions in this paper are summarized as follows:

  • Considering the cognitive spectrum sharing in cooperative networks,Footnote 1 we take into account the scarcity of the spectrum utilization between the small-cell transmitters and receivers. On the other hand, the wireless backhaul links are modeled with its nature unreliability,Footnote 2 where we employ the Bernoulli process in the system model.

  • In order to maximize the received signal-to-noise (SNR) ratio at the receivers, we employ the SC protocol, in which the perfect CSI are unnecessary at the transmitters [26, 27]. Moreover, the Nakagami-m fading is used to model the communication and interference channels since it provides various empirical scenarios for simulation [28].

  • The transmit power at each transmitter is practically formulated, where the peak interference threshold at the primary user and the maximal allowance transmit power to the secondary user are taken into account [17, 18]. We define S-SNR as the end-to-end SNR at the secondary receiver, which is the product of backhaul reliability random process and the distribution process of channels between the transmitters and the secondary receiver.

  • Based on the derived statistics of the S-SNR of the proposed systems, we derive the closed-form expressions of the outage probability, ergodic capacity and the symbol error rate along with the asymptotic expressions in high-SNR regime. Thus, the analytical results are validated using Monte Carlo simulation.

The remainder of this paper is organized as follows. In the next section, we first detail the system channel model of the proposed cognitive networks in the cooperative systems. In Section 3, the statistical properties of the S-SNR are derived under the existing of backhaul unreliability and the transmitter power constraints. In Section 4, the closed-form expressions of the outage probability, ergodic capacity and symbol error rate and its asymptotic performance are presented. Simulation results are presented in Section 5 and conclusions are drawn in Section 6.

Notation: \(\mathcal {CN}(\mu ,{\sigma _{n}^{2}})\) denotes the complex Gaussian distribution with mean μ and variance \({\sigma _{n}^{2}}\); F λ (γ) and f λ (γ) denote the cumulative distribution (CDF) and probability density function (PDF) of the random variable (RV) λ, respectively; \(\mathbb {E}_{\lambda }\left \lbrace f(\gamma ) \right \rbrace \) denotes the expectation of f(γ) with regard to the RV λ. In addition, \(\binom {\tau _{1}}{\tau _{2}}= \frac {\tau _{1}!}{\tau _{2}!(\tau _{1}-\tau _{2})}\) denotes the binomial coefficient. The other notations are listed in Table 1.

Table 1 Notation Used in this paper
Full size table

2 System and channel models

As illustrated in Fig. 1, we consider a cognitive network in a cooperative spectrum sharing system consisting of a macro base station (macro-BS) which is connected to the backbone network, K small-cells {SC1,...,SC k ,...,SC K } as the secondary network transmitters (SU-T k ) are connected to the macro-BS via unreliable wireless backhaul links, one secondary receiver (SU-D) and one primary user (PU-P). The K transmitters communicate with the secondary receiver SU-D by sharing the same spectrum with the primary user PU-P. We assume the perfect CSI for SU-PU channels can be obtained at the secondary transmitters i.e., small-cell transmitters. All nodes are assumed to be equipped with a single antenna and operate in half-duplex mode.

Fig. 1
figure 1

A cognitive HetNet with K cooperative transmitters, each communicates with the secondary user SU-D by employing cognitive spectrum sharing with the primary user PU-P, with the wireless backhaul links between those transmitters and macro-BS being unreliable

Full size image

In the practical systems, the transmit powers at each transmitter SU-T k are constrained due to the interference of the secondary network and must not exceed the peak interference power \(\mathcal {I}_{p}\) at the receiver PU-P. In addition, each transmitter is allowed to transmit up to their maximum power \(\mathcal {P}_{T}\) [17, 18, 29]. Under the combined power constraints, the transmit power at the transmitter SU-T k can be mathematically written as [13, 17]

$$ \tilde{P}_{k} = \min \left( \mathcal{P}_{T},\frac{\mathcal{I}_{p}}{|{h_{k}^{p}}|^{2}} \right). $$
(1)

where \({h_{k}^{p}}, k \in \lbrace 1,2,...,K \rbrace \) denotes the channel coefficients of the interference links SU-T k → PU-P. Recall that \(\mathcal {I}_{p}\) denotes the peak interference power at the receiver PU-P [30]. Without considering the backhaul reliability, the S-SNR over the channel from the transmitter SU-T k to the receiver SU-D is given as

$$ {\gamma_{k}^{s}} = \min \left( \bar{\gamma}_{\mathcal{P}}|{h_{k}^{s}}|^{2}, \frac{\bar{\gamma}_{\mathcal{I}}}{|{h_{k}^{p}}|^{2}}|{h_{k}^{s}}|^{2} \right), $$
(2)

where \({h_{k}^{s}}, k \in \lbrace 1,2,...,K \rbrace \) denotes the channel coefficients of the communication links SU-T k → SU-D. The average SNR of the primary and secondary network is given as \(\bar {\gamma }_{\mathcal {I}} = \mathcal {I}_{p}/{\sigma _{n}^{2}}\) and \(\bar {\gamma }_{\mathcal {P}} = \mathcal {P}_{T}/{\sigma _{n}^{2}}\) , respectively, with \({\sigma _{n}^{2}}\) representing the noise variance.

Due to the unreliable nature backhaul links, the signal received at the receiver SU-D via the transmitter SU-T k is given by

$$ r^{k,s}=\sqrt{\tilde{P}_{k}}({h_{k}^{s}})(\mathbb{I}_{k})x+n^{k,s}, $$
(3)

where \(\tilde {P}_{k}\) recalls the combined constraints transmit power at the transmitter SU-T k and \(n^{k,s}\sim \mathcal {CN}(0,{\sigma _{n}^{2}})\). Since the message is tranmitted from the core network to the receiver, it must go through the backhaul links and perform the success/failure transmission due to the characteristic of wireless links. Thus, the backhaul reliability \(\mathbb {I}_{k}\) of the transmitter SU-T k is modeled as Bernoulli process [7] with successful probability {Λ k ,∀k}, i.e., the SU-T k will successfully receive the message from macro-BS and forward to the receiver SU-D. Otherwise, the transmitter SU-T k does not send anything with failure probability being (1 −Λ k ). We denote x as the desired symbol transmitted by the small-cell transmitters and assume that \(\mathbb {E} \lbrace x \rbrace = 0\) and \(\mathbb {E} \lbrace |x|^{2} \rbrace = 1\).

Herein, we assume the SC protocol at the receiver SU-DFootnote 3 by selecting the small-cell station which has the best SNR over the received signals from K transmitters. Upon applying the SC protocol, it can be defined as

$$ k^{*} = \max \arg_{[k \in K]}({\gamma_{k}^{s}} \mathbb{I}_{k}), $$
(4)

is the selected transmitter SU-T k index. Consequently, the instantaneous S-SNR at the receiver SU-D can be obtained as

$$ \gamma_{S} = \min \left( \bar{\gamma}_{\mathcal{P}}|h_{k^{*}}^{s}|^{2}, \frac{\bar{\gamma}_{\mathcal{I}}}{|h_{k^{*}}^{p}|^{2}}|h_{k^{*}}^{s}|^{2} \right) \mathbb{I}_{k^{*}} . $$
(5)

As can be seen from Eq. 5, the end-to-end SNR is decided by the unreliable backhaul of the considered HetNets, i.e., the Bernoulli RV \(\mathbb {I}_{k}\). In addition, we assume all channels undergo Nakagami-m fading, i.e., a set of channel coefficients \(\lbrace {h^{s}_{k}}, \forall k \rbrace \) of the links SU-T k → SU-D and a set of channels \(\lbrace {h^{p}_{k}}, \forall k \rbrace \) of the links SU-T k → PU-P are distributed according to the gamma distribution, which is denoted by \(|{h_{k}^{s}}|^{2} \sim \text {Ga}({\mu _{k}^{s}},{\eta _{k}^{s}})\) and \(|{h_{k}^{p}}|^{2} \sim \text {Ga}({\mu _{k}^{p}},{\eta _{k}^{p}})\), respectively. Hence, The PDF and CDF of the RV χ ∼Ga(μ χ ,η χ ), where \(\chi \in \lbrace {h_{k}^{s}}, {h_{k}^{p}} \rbrace \) are given, respectively, as [18]

$$\begin{array}{@{}rcl@{}} &&f_{\chi}(x)=\frac{1}{(\mu_{\chi}-1)!(\eta_{\chi})^{\mu_{\chi}}}x^{\mu_{\chi}-1} e^{\left( -x/\eta_{\chi}\right)}, \\ &&F_{\chi}(x)=\left( 1-e^{\left( -x/\eta_{\chi}\right)}\sum\limits_{i=0}^{\mu_{\chi}-1} \frac{1}{i!} \left( x/\eta_{\chi}\right)^{i} \right), \end{array} $$
(6)

where \(\mu _{\chi } \in \lbrace {\mu _{k}^{s}}, {\mu _{k}^{p}} \rbrace \) represents the positive fading severity parameter [34, 35] with channel powers \(\lbrace {{\Omega }_{k}^{s}},{{\Omega }_{k}^{p}} \rbrace \), and \(\eta _{\chi } \in \lbrace {\eta _{k}^{s}}={{\Omega }_{k}^{s}}/{\mu _{k}^{s}}, {\eta _{k}^{p}}={{\Omega }_{k}^{p}}/{\mu _{k}^{p}} \rbrace \) indicates the scale factor on the corresponding channel.

3 Closed-form statistics of S-SNR in cognitive heterogeneous systems

In this section, our challenges are how to derive the statistical properties of the S-SNR with respect to the backhaul reliability and the combined power constraints at SU-T k . Without loss of generality, we assume that all channels follow the independent and identically distributed (i.i.d.) Nakagami-m fading, i.e., \( \mu _{s} = {\mu _{k}^{s}}, \eta _{s} = {\eta _{k}^{s}}, {\Lambda }={\Lambda }_{k},\mathbb {I}=\mathbb {I}_{k},\forall k\in K \) for transmission signals respect to the receiver SU-D and \( \mu _{p} = {\mu _{k}^{p}}, \eta _{p} = {\eta _{k}^{p}} ,\forall k\in K \) for the interference signals at the receiver PU-P, respectively. We first obtain the CDF of S-SNR for the signal between the particular SU-T k and SU-D, which is given in the following lemma

Lemma 1

For a cognitive HetNet with unreliable backhaul links, where transmitter SU-T k utilizes the sharing spectrum with the primary user PU-P, the CDF of the S-SNR for particular transmitter, \({\gamma _{k}^{s}}\) , is given as

$$\begin{array}{@{}rcl@{}} F_{\gamma_{k}^s \mathbb{I}_k}(x)= 1- {\Lambda} ({\Theta}_{1}(x) +{\Theta}_{2}(x)), \end{array} $$
(7)

where \({\Phi } = \frac {\Upsilon \left (\mu _{p} ,\bar {\gamma }_{\mathcal {I}} /\bar {\gamma }_{\mathcal {P}} \eta _{p}\right )}{\Gamma (\mu _{p})} \) , \(\epsilon = \frac {\bar {\gamma }_{\mathcal {I}} \eta _{s}}{\eta _{p}}\) and

$$\begin{array}{@{}rcl@{}} &&{\Theta}_{1}(x) = {\Phi} e^{-\left( x/\bar{\gamma}_{\mathcal{P}} \eta_{s}\right)} \sum\limits_{i=0}^{\mu_{s}-1} \frac{1}{i!}\left( x/\bar{\gamma}_{\mathcal{P}} \eta_{s}\right)^{i} , \\ &&{\Theta}_{2}(x) = \sum\limits_{j=0}^{\mu_{s}-1} \sum\limits_{g=0}^{\mu_{p}+j-1} \binom{\mu_{p}+j-1}{\mu_{p}-1} \frac{1}{g!(\bar{\gamma}_{\mathcal{P}} \eta_{s})^{g}} \epsilon^{\mu_{p}} e^{-\left( \bar{\gamma}_{\mathcal{I}}/\bar{\gamma}_{\mathcal{P}} \eta_{p} \right)} \frac{x^{j}e^{- \left( x/\bar{\gamma}_{\mathcal{P}} \eta_{s}\right)} \left( x + \epsilon \right)^{g}}{\left( x + {\epsilon} \right)^{\mu_{p} +j}}. \end{array} $$
(8)

Proof

The proof is given in Appendix A. □

In Eq. 7, Γ(.) and Υ(.,.) are the Gamma function [36, Eq. (8.310.1)] and the lower incomplete Gamma function [36, Eq. (8.350.1)], respectively. Next, the corresponding CDF and PDF for the received S-SNR at the receiver SU-D will be derived in the following theorem.

Theorem 1

For the i.i.d. Nakagami- m fading channels between K cooperative transmitters and the secondary receiver SU-D in the cognitive spectrum sharing withthe primary user PU-P, the CDF of the RV \(\gamma _{S} \overset {\triangle }{=} \max ({\gamma _{1}^{s}} \mathbb {I}_{1},..., {\gamma _{K}^{s}} \mathbb {I}_{K} )\) with respect to SC protocol and unreliable backhaul links is given by Eq. 9 in the top of next page.

$$ F_{\gamma_{S}}(x) = 1 + \sum\limits_{k=1}^{K} \binom{K}{k} (-1)^{k} \widehat{\sum\limits_{k,\mu_{s},\mu_{p},{\Lambda},{\Phi}}} \frac{x^{\widetilde{\varphi_{1}}} e^{-\beta x}} {(x+\epsilon)^{\widetilde{\varphi_{2}}}} , $$
(9)

where \(\widetilde {L_{a_{n}}}\)isdefined as \(\widetilde {L_{a_{n}}} \overset {\triangle }{=} {\sum }_{b_{n}=0}^{\mu _{p}+n-2} b_{n}a_{b_{n} +1}\;\),\(\beta \overset {\triangle }{=} k/\bar {\gamma }_{\mathcal {P}}\eta _{s}, \widetilde {\varphi _{1}} \overset {\triangle }{=} {\sum }_{\vartheta =0}^{\mu _{s}-1}\vartheta u_{\vartheta +1} + {\sum }_{t=0}^{\mu _{s}-1} t w_{t+1} +c_{1}+c_{2}+...+c_{\mu _{s}}\;\),\(\widetilde {\varphi _{2}} \overset {\triangle }{=} {\sum }_{t=0}^{\mu _{s}-1}(\mu _{p}+t)w_{t+1}\;\)and\(\widehat {\sum \limits _{k,\mu _{s},\mu _{p},{\Lambda },{\Phi }}}\)is ashorthand notation of

$$\begin{array}{@{}rcl@{}} \widehat{\sum\limits_{k,\mu_{s},\mu_{p},{\Lambda},{\Phi}}} &\overset{\triangle}{=}& \sum\limits_{l=0}^{k}\binom{k}{l} \sum\limits_{u_{1}...u_{\mu_{s}}}^{k-l}\sum\limits_{w_{1}...w_{\mu_{s}}}^{l}\sum\limits_{a_{1,1}...a_{1,\mu_{p}}}^{w_{1}} \sum\limits_{a_{2,1}...a_{2,\mu_{p}+1}}^{w_{2}} ... \sum\limits_{a_{\mu_{s},1}...a_{\mu_{s},\mu_{p} + \mu_{s}-1}}^{w_{\mu_{s}}} \frac{(k-l)!}{u_{1}!...u_{\mu_{s}}!} \frac{l!}{w_{1}!...w_{\mu_{s}}!} \\ && \frac{w_{1}!}{a_{1,1}!...a_{1,\mu_{p}}!} \frac{w_{2}!}{a_{2,1}!...a_{2,\mu_{p}+1}!} ... \frac{w_{\mu_{s}}!}{a_{\mu_{s},1}!...a_{\mu_{s},\mu_{p} + \mu_{s}-1}!} \prod\limits_{t=0}^{\mu_{s}-1} {\binom{\mu_{p}+t-1}{\mu_{p}-1}}^{w_{t+1}} \frac{1}{{\prod}_{\vartheta=0}^{\mu_{s}-1}\left( \vartheta!(\bar{\gamma}_{\mathcal{P}}\eta_{s})^{\vartheta} \right)^{u_{\vartheta+1}}} \\ & & \frac{1}{{\prod}_{b_{1}=0}^{\mu_{p}-1}\left( b_{1}!(\bar{\gamma}_{\mathcal{P}}\eta_{s})^{b_{1}} \right)^{a_{1,b_{1}+1}}} \frac{1}{{\prod}_{b_{2}=0}^{\mu_{p}}\left( b_{2}!(\bar{\gamma}_{\mathcal{P}}\eta_{s})^{b_{2}} \right)^{a_{2,b_{2}+1}}} ...\frac{1}{{\prod}_{b_{\mu_{s}}=0}^{\mu_{p}+\mu_{s}-2}\left( b_{\mu_{s}}!(\bar{\gamma}_{\mathcal{P}}\eta_{s})^{b_{\mu_{s}}} \right)^{a_{\mu_{s},b_{\mu_{s}}+1}}} \\ && \sum\limits_{c_{1}=0}^{\widetilde{L_{a_{1}}}} \sum\limits_{c_{2}=0}^{\widetilde{L_{a_{2}}}} ...\sum\limits_{c_{\mu_{s}}=0}^{\widetilde{L_{a_{\mu_{s}}}}} \binom{\widetilde{L_{a_{1}}}}{c_{1}} \binom{\widetilde{L_{a_{2}}}}{c_{2}}... \binom{\widetilde{L_{a_{\mu_{s}}}}}{c_{\mu_{s}}} {\Lambda}^{k} {\Phi}^{k-l} e^{-\left( \bar{\gamma}_{\mathcal{I}} l/{\bar{\gamma}_{\mathcal{P}} \eta_{p}} \right)} \epsilon^{\left( \widetilde{L_{a_{1}}}+\widetilde{L_{a_{2}}}+...+ \widetilde{L_{a_{\mu_{s}}}} +\mu_{p} l -(c_{1}+c_{2}+...+c_{\mu_{s}}) \right)} . \end{array} $$
(10)

Proof

The proof is given in Appendix B. □

Hence, the PDF of the received S-SNR can be derived as follows

$$ f_{\gamma_{S}}(x)= \sum\limits_{k=1}^{K} \binom{K}{k} (-1)^{k} \widehat{\sum\limits_{k,\mu_{s},\mu_{p},{\Lambda},{\Phi}}} \frac{ e^{-\beta x}}{(x+\epsilon)^{\widetilde{\varphi_{2}}+1}} \left( (\widetilde{\varphi_{1}}-\widetilde{\varphi_{2}}-\epsilon \beta)x^{\widetilde{\varphi_{1}}} + \widetilde{\varphi_{1}} \epsilon x^{\widetilde{\varphi_{1}}-1} -\beta x^{\widetilde{\varphi_{1}}+1} \right). $$
(11)

Remark 1

The statistics for S-SNR are different from those in existing works such as [7, 8, 17, 31] since the cognitive spectrum sharing and the Bernoulli process are taken into account. Theorem 1 completely characterizes the S-SNR of the proposed cooperative communications for cognitive HetNets. As a result, it is applicable to extend to special non-cooperative and Rayleigh fading scenarios. Hence, we will utilize the results in theorem 1 to derive the performance metrics, as well as the interesting scenarios in the following section.

4 Performance analysis of the proposed cognitive HetNets

In this section, we present the exact formulas of the important performance metrics such as outage probability, ergodic capacity and symbol error rate based on the statistics derived in Section 3. In order to get further insights, we will provide the scaling results for the asymptotic performance in the high-SNR regime.

4.1 Outage probability analysis

To investigate the performance of the proposed cognitive HetNets with unreliable backhaul connections over i.i.d. Nakagami-m fading channels, we first focus on the outage probability. Given a certain SNR threshold γ th, the outage probability of the S-SNR is defined as the probability that the S-SNR is below the threshold γ th, which can be written as

$$ \mathcal{P}_{out}(\gamma_{\text{th}}) \overset{\triangle}{=} \Pr\left( \gamma_{S} \leq \gamma_{\text{th}} \right) = F_{\gamma_{S}}(\gamma_{\text{th}}). $$
(12)

In other words, the outage probability can be expressed as the CDF of the S-SNR at the given γ th. By substituting (9) into (12), the outage probability is derived in the following theorem.

Theorem 2

The outage probability closed-form expression for the proposed cognitive HetNets with respect to the unreliable backhaul links is derived as

$$ \mathcal{P}_{out}(\gamma_{\text{th}}) = 1 + \sum\limits_{k=1}^{K} \binom{K}{k} (-1)^{k} \widehat{\sum\limits_{k,\mu_{s},\mu_{p},{\Lambda},{\Phi}}} \frac{\gamma_{\text{th}}^{\widetilde{\varphi_{1}}} e^{-\beta \gamma_{\text{th}}}} {(\gamma_{\text{th}}+\epsilon)^{\widetilde{\varphi_{2}}}}. $$
(13)

In the following, we show the case of interests on the outage probability for non-cooperative and Rayleigh fading scenarios.

4.1.1 Non-cooperative scenario

Corollary 1

Considering K = 1, the outage probability of the non-cooperative system is given by

$$\begin{array}{@{}rcl@{}} \mathcal{P}_{out}^{non} (\gamma_{\text{th}}) &=& 1- {\Lambda} {\Phi} \frac{\Gamma \left( \mu_{s}, \gamma_{\text{th}}/\bar{\gamma}_{\mathcal{P}} \eta_{s} \right)}{\Gamma(\mu_{s})} \\ &&- {\Lambda} \sum\limits_{j=0}^{\mu_{s}-1} \frac{\epsilon^{\mu_{p}} \gamma_{\text{th}}^{j} {\Gamma} \left( \mu_{p} +j, \frac{1}{\bar{\gamma}_{\mathcal{P}} \eta_{s}} (\gamma_{\text{th}} + \epsilon) \right)}{j! {\Gamma}(\mu_{p})\left( \gamma_{\text{th}} + \epsilon \right)^{\mu_{p}+j}} .\\ \end{array} $$
(14)

Proof

By extending from the CDF provided in Eq. 9 with the help of [36, Eq. (8.352.4)], we can derive the outage probability of non-cooperative scenario. □

4.1.2 Rayleigh fading scenario

In the Rayleigh Fading scenario, the channel fading severity of \({h_{k}^{s}}\) and \({h_{k}^{p}}\) are set to 1 i.e., μ s = μ p = 1, respectively. Therefore, the outage probability of the proposed system in Rayleigh fading is given as

$$\begin{array}{@{}rcl@{}} \mathcal{P}_{out}^{Ray}(\gamma_{\text{th}}) &=& 1+ \sum\limits_{k=1}^{K} \sum\limits_{l=0}^{k} \binom{K}{k} \binom{k}{l} (-1)^{k+l} \\ &&{\Lambda}^{k} e^{-\left( (k\gamma_{\text{th}}+\epsilon l)/\bar{\gamma}_{\mathcal{P}} \eta_{s} \right)} \left( \frac{\gamma_{\text{th}}}{\gamma_{\text{th}} + \epsilon} \right)^{l} . \end{array} $$
(15)

To provide insight into how the fading parameters and backhaul reliability impact the network performance, we next derive the asymptotic outage probability in the high-SNR regime of the considered system. In this case, we assume the peak interference threshold \(\bar {\gamma }_{\mathcal {I}}\) is proportional to the maximum transmit power \(\bar {\gamma }_{\mathcal {P}}\). The asymptotic outage probability is given in the following theorem as

Theorem 3

At the high-SNR regime with respect to \(\bar {\gamma }_{\mathcal {P}}\) as \(\bar {\gamma }_{\mathcal {P}} \rightarrow \infty \) in the cognitive sharing system with K cooperative transmitters and unreliable backhaul links, the asymptotic outage probability is given by

$$ \mathcal{P}_{out}^{Asy}(\gamma_{\text{th}}) \overset{\bar{\gamma}_{\mathcal{P}} \rightarrow \infty}{=} \left( 1-{\Lambda}\right)^{K} = {\Xi}. $$
(16)

Proof

The proof is given in Appendix C. □

Remark 2

Since the unreliable backhaul links exist, the asymptotic outage probability limitation is only determined by the reliability of backhaul links.

4.2 Ergodic capacity analysis

Ergodic capacity (nat/s/Hz) is defined as the statistical mean of the instantaneous SNR between the transmitters and receivers. Mathematically, the ergodic capacity can be derived as [19, 37]

$$ \mathcal{C} \overset{\triangle}{=} \mathbb{E}_{\gamma_{S}}\left\lbrace \log_{2}(1+x)\right\rbrace = {\int}_{0}^{\infty} \log_{2}(1+x) f_{{\gamma_{S}}}\left( {x}\right) dx. $$
(17)

Using the integration-by-part method, (17) can be written as

$$\begin{array}{@{}rcl@{}} \mathcal{C} &=& \frac{1}{\ln(2)}{\int}_{0}^{\infty} \frac{1}{1+x}(1-F_{{\gamma_{S}}}\left( {x}\right)) dx \\ &=& -\frac{1}{\ln(2)} \sum\limits_{k=1}^{K} \binom{K}{k} (-1)^{k} \widehat{\sum\limits_{k,\mu_{s},\mu_{p},{\Lambda},{\Phi}}} {\int}_{0}^{\infty} \frac{x^{\widetilde{\varphi_{1}}} e^{-\beta x}} {(1+x)(x+\epsilon)^{\widetilde{\varphi_{2}}}} dx.\\ \end{array} $$
(18)

Unfortunately, the integral in Eq. 18 cannot be evaluated in closed-form. However, it can be easily evaluated in numerical way since the integrand includes only elementary functions which is the built-in function in mathematical tools, e.g., Mathematica.

4.2.1 Non-cooperative scenario

Corollary 2

We fix K = 1, the ergodic capacity of the non-cooperative scenario is given as

$$\begin{array}{@{}rcl@{}} \mathcal{C}^{non} &=& \frac{\Lambda {\Phi}}{\ln(2)} \sum\limits_{i=0}^{\mu_{s}-1} \frac{\Gamma(i+1) {\Psi}(i+1;i+1;1/{\bar{\gamma}_{\mathcal{P}} \eta_{s}})}{i!\left( {\bar{\gamma}_{\mathcal{P}} \eta_{s}}\right)^{i}}\\ &&+ \frac{\Lambda}{\ln(2)} \sum\limits_{j=0}^{\mu_{s}-1} \sum\limits_{g=0}^{\mu_{p}+j-1} \sum\limits_{l=0}^{g} \binom{\mu_{p}+j-1}{\mu_{p}-1} \binom{g}{l}\\ &&\epsilon^{\mu_{p} +g-l} e^{-\left( \bar{\gamma}_{\mathcal{I}}/{\bar{\gamma}_{\mathcal{P}} \eta_{p}} \right)} \frac{1}{g!(\bar{\gamma}_{\mathcal{P}}\eta_{s})^{g}} {\int}_{0}^{\infty}\\ && \frac{x^{j+l}e^{- \left( x/{\bar{\gamma}_{\mathcal{P}}\eta_{s}}\right)}}{(1+x)\left( x + {\epsilon} \right)^{\mu_{p}+j}}dx. \end{array} $$
(19)

Proof

The C non is derived by expanding from Eq. 18 with K = 1 and the help of [38, Eq. (2.3.6.9)], where in Eq. 19, Ψ(a,b,c) denotes the confluent hypergeometric function [36, Eq. (9.211.4)], where \({\Psi } \left (a,b,c \right ) = \frac {1}{\Gamma (a)} {\int }_{0}^{\infty } e^{-ct}t^{b-a}(1+t)^{b-a-1}dt\). □

4.2.2 Rayleigh fading scenario

While μ s = μ p = 1, the ergodic capacity of the proposed system in Rayleigh fading scenario is given by

$$\begin{array}{@{}rcl@{}} \mathcal{C}^{Ray} &=&- \frac{1}{\ln(2)} \sum\limits_{k=1}^{K} \sum\limits_{l=0}^{k} \binom{K}{k} \binom{k}{l} (-1)^{k+l} \\ &&{\Lambda}^{k} e^{-(\epsilon l/\bar{\gamma}_{\mathcal{P}} \eta_{s})} {\int}_{0}^{\infty} \frac{x^{l} e^{- \left( k x/{\bar{\gamma}_{\mathcal{P}} \eta_{s}}\right)}} {(1+x)\left( x + {\epsilon} \right)^{l}}dx. \end{array} $$
(20)

4.3 Symbol error rate analysis

The symbol error rate is considered as critical performance metrics in wireless system. For most modulation schemes, the symbol error rate of the S-SNR can be evaluated as [19, 37]

$$ P_{e} \quad \overset{\triangle}{=} \quad \frac{A \sqrt{B}}{2\sqrt{\pi}} {\int}_{0}^{\infty} x^{-1/2}e^{-Bx}F_{{\gamma_{S}}}\left( {x}\right)dx, $$
(21)

where (A,B) are the constants determined by the specific modulation schemes. Now applying the CDF function \(F_{\gamma _{S}}({x})\) which is given by Eqs. 9 into 21, we can derive the corresponding expression in the following

Corollary 3

The symbol error rate of the proposed system can be expressed as

$$\begin{array}{@{}rcl@{}} P_{e}&=& \frac{A}{2} + \frac{A \sqrt{B}}{2\sqrt{\pi}} \sum\limits_{k=1}^{K} \binom{K}{k} (-1)^{k} \widehat{\sum\limits_{k,\mu_{s},\mu_{p},{\Lambda},{\Phi}}} \epsilon^{\widetilde{\varphi_{1}}+\frac{1}{2}-\widetilde{\varphi_{2}}} {\Gamma} \\ &&\left( \widetilde{\varphi_{1}}+\frac{1}{2}\right) {\Psi} \left( \widetilde{\varphi_{1}}+\frac{1}{2}; \widetilde{\varphi_{1}}+\frac{3}{2}-\widetilde{\varphi_{2}}; \epsilon (\beta +B) \right).\\ \end{array} $$
(22)

Proof

The proof is given in Appendix D. □

4.3.1 Non-cooperative scenario

Corollary 4

For non-cooperative scenario K = 1, the average symbol error rate is given as

$$\begin{array}{@{}rcl@{}} P_{e}^{non} &=&\frac{A}{2} - \frac{A \sqrt{B}}{2\sqrt{\pi}} {\Lambda} {\Phi}\sum\limits_{i=0}^{\mu_{s}-1} \frac{1}{i!\left( {\bar{\gamma}_{\mathcal{P}} \eta_{s}}\right)^{i}} \frac{\Gamma(i+1/2)}{(\beta+B)^{i+1/2}}\\ &&- \frac{A \sqrt{B}}{2\sqrt{\pi}}{\Lambda} \sum\limits_{j=0}^{\mu_{s}-1} \sum\limits_{g=0}^{\mu_{p}+j-1} \sum\limits_{l=0}^{g} \binom{\mu_{p}+j-1}{\mu_{p}-1} \\ && \binom{g}{l} \frac{ e^{-\left( \bar{\gamma}_{\mathcal{I}}/{\bar{\gamma}_{\mathcal{P}} \eta_{p}} \right)}}{g!(\bar{\gamma}_{\mathcal{P}} \eta_{s})^{g}} \epsilon^{g+1/2} {\Gamma}(j+l+1/2) \\ && {\Psi}(j+l+1/2; l+3/2-\mu_{p};\left( 1/{\bar{\gamma}_{\mathcal{P}}\eta_{s}} +B\right)\epsilon). \end{array} $$
(23)

Proof

The \(P_{e}^{non}\) is derived by expanding from Eq. 9 with K = 1 and the help of [38, Eq. (2.3.6.9)] □

4.3.2 Rayleigh fading scenario

Setting μ s = μ p = 1, the average symbol error rate of the proposed system in Rayleigh fading scenario is given by

$$\begin{array}{@{}rcl@{}} P_{e}^{Ray} &=& \frac{A}{2} + \frac{A \sqrt{B}}{2\sqrt{\pi}}\sum\limits_{k=1}^{K} \sum\limits_{l=0}^{k} \binom{K}{k} \binom{k}{l} (-1)^{k+l} {\Lambda}^{k} e^{-\epsilon l/\bar{\gamma}_{\mathcal{P}} \eta_{s}} \epsilon^{1/2}\\&& {\Gamma}(l+1/2) {\Psi}(l+1/2;3/2; (k/\bar{\gamma}_{\mathcal{P}}\eta_{s} +B) \epsilon). \end{array} $$
(24)

Theorem 4

The asymptotic of the average symbol error rate as \(\bar {\gamma }_{\mathcal {P}} \rightarrow \infty \) of the proposed cognitive cooperative system is given by

$$ P_{e}^{Asy} \overset{\bar{\gamma}_{\mathcal{P}} \rightarrow \infty}{=} \left( 1-{\Lambda} \right)^{K} = {\Xi}. $$
(25)

Proof

The proof is similar to [31]. According to Eq. 41, the CDF of S-SNR is expressed as

$$\begin{array}{@{}rcl@{}} F_{\gamma_{S}}(x) &=& \left[ 1- {\Lambda} {\Phi} \frac{\Gamma\left( \mu_{s},\frac{x}{\bar{\gamma}_{\mathcal{P}} \eta_{s}}\right)}{\Gamma(\mu_{s})}\right.\\ &&\left.- {\Lambda} \sum\limits_{j=0}^{\mu_{s}-1}\frac{\epsilon^{\mu_{p}} x^{j} {\Gamma}\left( \mu_{p}+j,\frac{x+\epsilon} {\bar{\gamma}_{\mathcal{P}} \eta_{s}} \right)}{j!{\Gamma}(\mu_{p}) \left( x+\epsilon\right)^{\mu_{p}+j}} \right]^{K}. \end{array} $$
(26)

Since \(\bar {\gamma }_{\mathcal {P}} \rightarrow \infty \), \({\sum }_{j=0}^{\mu _{s}-1}(.)\) is dominated with j = 0 , and \({\Gamma }\left (\mu _{\chi },x/y \right ) \overset {y \rightarrow \infty }{\approx } {\Gamma }\left (\mu _{\chi } \right ) \) we have

$$ P_{e}^{Asy} \overset{\bar{\gamma}_{\mathcal{P}} \rightarrow \infty}{=} F_{\gamma_{S}}(x) \overset{\bar{\gamma}_{\mathcal{P}} \rightarrow \infty}{=} \left( 1- {\Lambda} \right)^{K}. $$
(27)

□

Remark 3

It can be observed from Theorem 4 that the asymptotic symbol error rate is not affected by Nakagami-m fading severity parameters. The average symbol error rate is converged to the same limitation as the outage probability with the same settings of the degrees of transmitter cooperation and the fading severity parameters.

5 Numerical results and discussions

In this section, we present the numerical results of the outage probability, ergodic capacity and symbol error rate to verify the analysis under the impact of unreliable backhaul links, the fading severity of primary and secondary networks. We also assume the SC protocol is perfectly performed in the simulations. The ”Sim” curves indicate the link-level Monte Carlo simulation results, while the ”Ana” and ”Asy” curves represent the analytical results and asymptotic performance at high-SNR regime, respectively.

We fix the S-SNR threshold γ th = 3 dB in the computation of the outage probability. Without loss of generality, we assume that the secondary user SU-D and the primary user PU-P are located at point [0,0] and [0.5,0.5], respectively. Those small-cell transmitters SU-T k are located at [0,0.5]. Hence, the channel mean powers are calculated by \({{\Omega }_{k}^{s}} = {{\Omega }_{k}^{p}} =\left (\sqrt {(x_{k}-x_{u})^{2}+(y_{k}-y_{u})^{2}}\right )^{\zeta }\), where u ∈{D,P} and ζ = 4 as the path-loss exponent. In this setting, we obtain the mean power of all links is equal to 16. We also assume the ratio of the interference power \(\bar {\gamma }_{\mathcal {I}}\) and the maximum transmit power \(\bar {\gamma }_{\mathcal {P}}\) is constant for all numerical results. We define the average SNR as \(\bar {\gamma } = \bar {\gamma }_{\mathcal {P}}\).

5.1 Outage probability analysis

Figures 2, 3 and 4 show the outage probability for various scenarios. In Fig. 2, we verify the accurate of the derived analytical outage probability versus the average SNR with the simulation. Assuming (Λ1 = 0.98,Λ2 = 0.98,Λ3 = 0.98) for K = 1, K = 2, K = 3, respectively. The fading severity parameters are initialized as \(\mu _{\chi } = \lbrace {\mu _{k}^{s}}=1,{\mu _{k}^{p}}=1 , \forall k\rbrace \). From this figure, it can be observed that all curves converge to the asymptotic limitation as \(\bar {\gamma }\) increases. Furthermore, the outage probability values get lower when more transmitters cooperate due to the correlation of multiple signals at the receiver SU-D.

Fig. 2
figure 2

Outage probability for various level of the degree of transmitter cooperation with fixed unreliable backhaul links

Full size image
Fig. 3
figure 3

Outage probability for various level of backhaul unreliability with fixed asymptotic limitation

Full size image
Fig. 4
figure 4

Outage probability for various Nakagami-m fading severity with Ξ = 1.44E-4

Full size image

To investigate the outage probability behavior at the same asymptotic threshold when the degree of transmitter cooperation is changed, we show it in Fig. 3. Assuming Ξ = 6.4E-5, we set (Λ1 = 0.999936), (Λ1 = 0.992,Λ2 = 0.992), and (Λ1 = 0.96,Λ2 = 0.96,Λ3 = 0.96) for case K = 1, K = 2, K = 3, respectively. The fading severity parameters are similar as in Fig. 2. We can observe that at the same outage probability asymptotic limitation, the higher degrees of transmitter cooperation converge faster than the others. Moreover, at the same degrees of transmitter cooperation (K = 1 or K = 3), the outage probability performance gets worst if the backhaul links is more unreliable, otherwise, the receiver SU-D performs the good performance.

Figure 4 plots the outage probability with various Nakagami-m fading severity scenarios at the fixed value (K = 2,Λ = 0.988). From these curves, it can be seen that the outage probability is strongly affected by the fading severity of the secondary network μ s rather than the primary network fading severity μ p . Specifically, the performance at the receiver SU-D tends to be better with the increase of μ s while the outage probability values seem unchanged with the alternation of μ p .

5.2 Ergodic capacity analysis

In Fig. 5, these curves illustrate the ergodic capacity for various scenarios in non-cooperative system. At fixed Λ = 0.999936, this figure shows that the capacity of (μ s = 2,μ p = 1) is nearly double increased compare to (μ s = 1,μ p = 2) with respect to the Rayleigh fading scenario (μ s = 1,μ p = 1). In other words, the increase of Nakagami-m fading severity both lead to the improvement of performance. However, the enhancement of Nakagami-m fading severity of the secondary network μ s significantly results in a high capacity rather than the fading severity of the primary network μ p . On the other hands, as the backhaul links tend to be more reliable, the achievable capacity at the receiver SU-D is increased if we compare the two particular scenarios (Λ = 0.8) and (Λ = 0.999936) at fixed (μ s = μ p = 1).

Fig. 5
figure 5

Ergodic capacity for various scenarios in non-cooperative system

Full size image

In Fig. 6, we plot the achievable capacity with various degrees of transmitter cooperation at fixed Ξ = 6.4E-5 and (μ s = μ p = 2) in cooperative systems. This plot shows that the degree of transmitter cooperation and backhaul links reliability are highly impact to the achievable capacity due to the increasing received SNR at SU-D. It can be said that if either more transmitters jointly cooperate or the backhaul links are more reliable, the capacity of the proposed system will be effectively improved.

Fig. 6
figure 6

Ergodic capacity for various degree of cooperation and backhaul reliability at fixed Ξ = 6.4E-5

Full size image

The various fading severity scenarios at fixed of (K = 2,Λ = 0.95) are shown in Fig. 7. It can be seen that the increasing of primary fading severity leads to a reduction in the achievable capacity, i.e., the capacity of (μ s = 1,μ p = 3) is lower than the capacity of (μ s = 1,μ p = 2). While in the case of increasing the μ s value, the achievable capacity of (μ s = 3,μ p = 1) is well performing compared to the capacity of (μ s = 2,μ p = 1).

Fig. 7
figure 7

Ergodic capacity for various channel fading severity

Full size image

5.3 Symbol error rate analysis

In Fig. 8, we show the correctness of the symbol error rate analysis in the proposed system compare to the simulation. Binary Phase-shift Keying (BPSK) is used as the signal constellation. We set (μ s = μ p = 2) and (Λ1 = 0.95,Λ2 = 0.95,Λ3 = 0.95) for K = 1, K = 2, K = 3, respectively. From the figure, we can see the exact matches between the analytically derived curves and the simulation curves for the symbol error rate. Moreover, it can be observed that all curves converge to the asymptotic limitations, which is similar to the asymptotic analysis in Theorem 4.

Fig. 8
figure 8

Symbol error rate for various degree of transmitter cooperation at fixed K = 2,μ s = μ p = 2

Full size image

In Figs. 9 and 10, we investigate the impact of various Nakagami-m fading severity, degree of transmitter cooperation and backhaul reliability scenarios on the symbol error rate performance. From those plots, we can obtain some observations as follows

  • At fixed Nakagami-m fading severity μ p of (μ s = μ p = 1), (μ s = 2,μ p = 1) and (μ s = 3,μ p = 1) , the higher μ s fading severity parameters result in lower symbol error rate than the others.

  • At fixed Nakagami-m fading severity μ s of (μ s = 2,μ p = 1), (μ s = μ p = 2) and (μ s = 2,μ p = 3) , the symbol error rate values are insignificantly changed with the increase of μ p .

  • The degree of transmitter cooperation and backhaul reliability are the key factors to reduce the symbol error rate in the considered system. Specifically, the symbol error rate significantly decreases if either backhaul links tend to be more reliable or the number of transmitters is increased.

Fig. 9
figure 9

Symbol error rate for various fading severity parameters at fixed Ξ = 2.5E-3

Full size image
Fig. 10
figure 10

Symbol error rate for various degree of transmitter cooperation and backhaul reliability

Full size image

6 Conclusions

In this paper, we have taken into account the cognitive HetNets with unreliable backhaul links over i.i.d. Nakagami-m fading. For those small-cell transmitters which utilize the same spectrum with primary user, their combined power constraints have been practically considered, i.e., the peak interference power at the primary user \(\mathcal {I}_{p}\) and the maximal allowance transmit power at each transmitter \(\mathcal {P}_{T}\). We have derived the closed-form expressions of the outage probability, ergodic capacity and symbol error rate as well as asymptotic performance to obtain study insights. It has been shown that the asymptotic performance is only determined by the unreliable backhaul links in the high-SNR regime. The performance of the proposed system is highly improved proportionally to the degree of cooperation and the fading severity of secondary network. Our analyzed results provide suitable framework for network designers to clearly understand the effects of unreliable backhaul links and decide whether enabling the CR networks for those cooperative transmitters in order to efficiently utilize the spectrum.