1 Introduction

With the increasing demands for various wireless services, the radio spectrum is becoming more and more crowded that needs to be utilized more efficiently. The US Federal Communications Commission (FCC) reported the utilization of the spectrum is usually less than 15 % [1]. To overcome this problem, the cognitive radio has emerged as a promising approach to improve the spectrum utilization efficiently. In a cognitive radio network, three basic spectrum sharing paradigms have been considered: i) interweave, ii) overlay and iii) underlay. In the interweave paradigm, the secondary (cognitive) users (SUs) opportunistically utilize the detected spectrum holes to avoid interference with the primary users (PUs). In the overlay paradigm, the SUs exploit the structure of the primary message and perform the interference precancelation (such as dirty paper pre-coding) for the non-intrusive SU transmissions [2]. In the underlay paradigm, the SU controls the transmission power over the operating bandwidth in a way that the SU signals may interfere with the primary signals within a tolerable limit [3, 4]. In this paper, we focus on the underlay cognitive network.

On the other hand, as a powerful spatial diversity technique, the cooperative relay communication has recently attracted a lot of attention to improve the performance over conventional point-to-point transmissions. There are originally two relay modes: amplify-and forward (AF) and decode-and-forward (DF) [5]. In the AF mode, the relay node amplifies the received signal and forwards that to the destination node. In this case, the noise is also amplified so that the system performance is limited. In the DF mode, the relay node decodes the source node’s signal and retransmits the signal to the destination node. While the system reliability can be improved in DF, the performance degrades when the relay node incorrectly decodes the received signals. Recently, a new hybrid relaying scheme that combines the merits of both of the relay modes has been proposed [6]. If the relay node decodes the message correctly, then it operates in DF mode. Otherwise, it will operate in the AF mode. In this paper, we consider all the three relay modes.

By combining the cognitive radio and the cooperative communication techniques, the cognitive relay network can significantly improve the spectrum utilization [7]. For a cognitive relay network, a fundamental question is how to effectively select the cooperative relay nodes. In the literature, a lot of research has been done on relay station network, without enough consideration for the user relay scenario [8, 9]. Cooperative beamforming for dual-hop amplify and forward cognitive relay networks was considered in [10], which aimed at improving the secondary system performance with limited feedback from the primary receiver. Under the context of the cognitive radio, the optimal power allocation schemes were derived in [11] for the outage capacity under both peak and average interference constraints. Reference [12] derived the capacity limits of the cognitive radio networks under the hybrid relay. In [13], the performance of underlay selective DF relay networks with non-necessarily identical fading parameters was evaluated for the Rayleigh fading channels. Without considering the relay selection, the performance analysis of the hybrid relay can be found in [1416]. In [17], the problem of spectrum sharing together with the adaptive user cooperation was investigated in heterogeneous cognitive relay systems.

This paper considers three different relay modes in an underlaid cognitive network. We aim to minimize the outage probability under both the peak and average interference constraints. The optimal relay selection can be determined by minimizing the outage probability. Moreover, we also discuss the performance of three different relaying techniques in terms of the interference temperature (INR) at the primary receiver.

The rest of the paper is organized as follows: Section 2 briefly describes the system model. In Section 3, for each of the relay mode, we derive the optimal power allocation to achieve the minimum outage probability under both the peak and the average interference power constraints. We also provide simulation results in Section 4 to validate our analysis. Finally, conclusions are drawn in Section 5.

2 System model

We consider a cooperative relay-based cognitive radio network, as shown in Fig. 1. In the primary network, one base station (BS) transmits directly to one PU. In the cognitive radio network, a cognitive user S U 0 communicates with the destination Access Point (AP) through a direct link and through K other SUs (S U k ) that potentially act as relays nodes.

Fig. 1
figure 1

The system model

Without the loss of generality, it is assumed that in a time slot t, the BS transmits a signal x p to the PU with a fixed power P B S . During the same time slot, the cognitive source node and the relay node transmit at two different stages. In Stage 1, the S U 0 transmits the signal x c with a unit energy to the relay node S U k and the AP. In Stage 2, the S U k forwards the signal received in Stage 1 to the AP using either the DF or AF. Finally, the AP combines the two received signal copies through the maximum-ratio-combining (MRC). During these two stages, the transmit power of the BS is fixed so that the interference to the cognitive users can be treated as either zero (when the SU is far away from the BS) or added as the additive white Gaussian noise (AWGN) [18] (when the SU is within the BS’s transmission range).

During the first stage, the received signals at the AP and S U k can be written as:

$$\begin{array}{@{}rcl@{}} Y_{0c}=\sqrt{P_{0}}h_{0c}x_{c}+n_{c}, \end{array} $$
(1)
$$\begin{array}{@{}rcl@{}} Y_{0k}=\sqrt{P_{0}}h_{0k}x_{c}+n_{k}, \end{array} $$
(2)

where P 0 is S U 0’s transmit power, and h 0c and h 0k denote the complex Gaussian channel gains between S U 0 and AP and S U 0 and S U k , respectively, n c and n k are the zero-mean AWGN with a variance of N. The signal to noise ratio (SNR) at the AP and the corresponding mutual information between the S U 0 and AP can be written as:

$$\begin{array}{@{}rcl@{}} SNR_{0}=\frac{P_{0}|h_{0c}|^{2}}{N}, \end{array} $$
(3)
$$\begin{array}{@{}rcl@{}} I_{0}=\log_{2}(1+SNR_{0}). \end{array} $$
(4)

During the second stage, with the AF as the relay, the signal x k transmitted by the S U k is given by:

$$\begin{array}{@{}rcl@{}} x_{k}=\sqrt{\frac{P_{k}}{P_{0}|h_{0k}|^{2}+N}}(\sqrt{P_{0}}h_{0k}x_{c}+n_{k}), \end{array} $$
(5)

where P k is the transmit power of S U k and \(\beta =\sqrt {\frac {P_{k}}{P_{0}|h_{0k}|^{2}+N}}\) is the amplifying factor. Consequently, the received signal at the AP terminal in the second stage can be represented by:

$$\begin{array}{@{}rcl@{}} Y_{kc}=h_{kc}x_{k}+n_{c}, \end{array} $$
(6)

where h k c is the complex Gaussian channel gain between S U k and the AP terminal. By combining Eqs. 5 and 6, the received SNR at the AP during the second stage becomes:

$$\begin{array}{@{}rcl@{}} SNR_{k}=\frac{P_{0}|h_{0c}|^{2}P_{k}|h_{kc}|^{2}}{N(P_{0}|h_{0c}|^{2}+P_{k}|h_{kc}|^{2}+N)}. \end{array} $$
(7)

Using the maximum-ratio-combining of Eqs. 3 and 7, the total SNR at the AP with AF as the relay is given by:

$$\begin{array}{@{}rcl@{}} SNR_{mrc,k}=\frac{P_{0}|h_{0c}|^{2}}{N}+\frac{P_{0}|h_{0c}|^{2}P_{k}|h_{kc}|^{2}}{N(P_{0}|h_{0c}|^{2}+ P_{k}|h_{kc}|^{2}+N)}. \end{array} $$
(8)

On the basis of the information theory for the AF relay, we can express the maximum average mutual information between S U 0 and the AP can be stared as:

$$\begin{array}{@{}rcl@{}} I_{AF}&=&\frac{1}{2}\log_{2}(1+SNR_{mrc,k})\\ &=&\frac{1}{2}\log_{2}\left( 1+\frac{P_{0}|h_{0c}|^{2}}{N}\right.\\ &&\qquad\quad\left.+ \frac{P_{0}|h_{0k}|^{2}P_{k}|h_{kc}|^{2}}{N(P_{0}|h_{0k}|^{2}+P_{k}|h_{kc}|^{2}+N)}\right). \end{array} $$
(9)

We can also rewrite Eq. 9 as:

$$\begin{array}{@{}rcl@{}} I_{AF}&=&\frac{1}{2}\log_{2}\left( 1+\frac{P_{0}|h_{0c}|^{2}}{N}+f\left( \frac{P_{0}|h_{0k}|^{2}}{N},\frac{P_{k}|h_{kc}|^{2}}{N}\right)\right),\\ \end{array} $$
(10)

where \(f(x,y)=\frac {xy}{1+x+y}\) is the Harmonic mean function that is exponentially distributed with the parameter \(\frac {N}{P_{0}|h_{0k}|^{2}}+\frac {N}{P_{k}|h_{kc}|^{2}}\) [19].

On the other hand, the relay node S U k can choose DF during the second stage. In such a case, the signal Y k c received at the AP terminal can be written as:

$$\begin{array}{@{}rcl@{}} Y_{kc}=\sqrt{P_{k}}h_{kc}x_{c}+n_{c} \end{array} $$
(11)

Similar to Eq. 4, the mutual information between the cognitive user S U 0 and the relay node S U k in the DF becomes:

$$\begin{array}{@{}rcl@{}} I_{0k}=\frac{1}{2}\log_{2}\left( 1+\frac{P_{0}|h_{0k}|^{2}}{N}\right). \end{array} $$
(12)

Consequently, the mutual information using DF as the relay becomes:

$$\begin{array}{@{}rcl@{}} I_{0kc}=\frac{1}{2}\log_{2}\left( 1+\frac{P_{0}|h_{0c}|^{2}}{N}+\frac{P_{k}|h_{kc}|^{2}}{N}\right). \end{array} $$
(13)

As shown in [20], the average mutual information can be represented by:

$$\begin{array}{@{}rcl@{}} I_{DF}&=&\frac{1}{2}\min\left\{\log_{2}\left( 1+\frac{P_{0}|h_{0k}|^{2}}{N}\right),\right.\\&&\left.\quad\qquad\log_{2} \left( 1+\frac{P_{0}|h_{0c}|^{2}}{N}+\frac{P_{k}|h_{kc}|^{2}}{N}\right)\right\}. \end{array} $$
(14)

3 Outage probability analysis and power allocation

To facilitate the performance analysis and comparisons, we derive closed-form expressions of the outage probability with three cooperative relaying protocols, namely AF, DF, and HDAF. For a general communication system, the outage occurs when the mutual information I falls below the target value R, as IR. Therefore, for a desired transmission rate R, the outage probability is defined as:

$$\begin{array}{@{}rcl@{}} P_{out}=Pr(I\leq R), \end{array} $$
(15)

where P r(⋅) denotes the outage probability. In particular, R satisfies:

$$\begin{array}{@{}rcl@{}} R=\log_{2}(1+\gamma_{th}), \end{array} $$
(16)

where γ t h denotes the SNR threshold.

During the two stages, the transmissions from the cognitive users cause interference with the PU. Now, let I p be the maximum interference level that can be tolerated by the PU. The peak interference constraints of the PU can be written as:

$$\begin{array}{@{}rcl@{}} P_{0}|h_{0p}|^{2}\leq I_{p}, \end{array} $$
(17)
$$\begin{array}{@{}rcl@{}} P_{k}|h_{kp}|^{2}\leq I_{p}. \end{array} $$
(18)

Therefore, an average interference constraint of the PU can be written as:

$$\begin{array}{@{}rcl@{}} \frac{P_{0}|h_{0p}|^{2}}{2}+\frac{P_{k}|h_{kp}|^{2}}{2}\leq I_{p}, \end{array} $$
(19)

where h 0p and h k p are the channel gains between S U 0 and PU and S U k and PU, respectively.

3.1 Amplify-and-forward relay transmission

If we assume that |h 0c | and |h 0k | are independent Rayleigh random variables, with the scale parameter as |H 0c | and |H 0k |, respectively, then in the AF relay mode the outage probability given in Eq. 15 can be calculated as:

$$\begin{array}{@{}rcl@{}} P_{out}=\frac{(\gamma_{th}N)^{2}}{2P_{0}|H_{0c}|^{2}}\left( \frac{1}{P_{0}|H_{0k}|^{2}}+\frac{1}{P_{k}|H_{0c}|^{2}}\right). \end{array} $$
(20)

Considering the peak interference constraint, we can see that Eq. 20 is a monotonic function for variables P 0 and P k . Therefore, the cognitive user S U 0 and the relay node S U k shall transmit at the maximum power. Form (17) and (18), the optimal power can be written as:

$$\begin{array}{@{}rcl@{}} P^{\ast}_{0}=\frac{I_{p}}{|h_{0p}|^{2}},\ P^{\ast}_{k}=\frac{I_{p}}{|h_{kp}|^{2}}. \end{array} $$
(21)

By considering the average interference constraint, the optimization problem can be formulated as:

$$\begin{array}{@{}rcl@{}} \min \ P_{out} \end{array} $$
$$\begin{array}{@{}rcl@{}} \text{subject to : the constraint} (19) \end{array} $$
(22)

It can be shown that the outage probability function in Eq. 20 is a convex optimization problem. Therefore, we can use Lagrange dual method to find the optimal solution. The Lagrangian function is given by:

$$\begin{array}{@{}rcl@{}} L(P_{0} , P_{k} , \lambda_{1})&=&-\frac{(\gamma_{th}N)^{2}}{2P_{0}|H_{0c}|^{2}}\left( \frac{1}{P_{0}|H_{0k}|^{2}}+ \frac{1}{P_{k}|H_{0c}|^{2}}\right)\\ &&-\lambda_{1}(P_{0}|h_{0p}|^{2}+P_{k}|h_{kp}|^{2}-2I_{p}), \end{array} $$
(23)

where λ 1 serves as the Lagrangian multiplier. According to

$$\begin{array}{@{}rcl@{}} \frac{\partial L}{\partial P_{0}}=0 ,\ \frac{\partial L}{\partial P_{k}}=0, \end{array} $$
(24)

and by using Eq. 24, the optimal value for \(P^{\ast }_{0}\) and \(P^{\ast }_{k}\) can be obtained with the AF as the relay:

$$\begin{array}{@{}rcl@{}} P^{\ast}_{0}=\frac{-b\pm\sqrt{b^{2}-4ac}}{2a}, \end{array} $$
$$\begin{array}{@{}rcl@{}} P^{\ast}_{k}=\frac{2I_{p}-P^{\ast}_{0}|h_{0p}|^{2}}{|h_{kp}|^{2}}, \end{array} $$
(25)

where

$$\begin{array}{@{}rcl@{}} a=2H^{2}_{0k}|h_{0p}|^{2}|h_{kp}|^{2}-2|H_{kc}|^{2}|h_{0p}|^{4}, \end{array} $$
$$\begin{array}{@{}rcl@{}} b=(8|H_{kc}|^{2}|h_{0p}|^{2}-2|H_{0k}|^{2}|h_{kp}|^{2})I_{p}, \end{array} $$
$$\begin{array}{@{}rcl@{}} c=-8|H_{kc}|^{2}{I^{2}_{p}}. \end{array} $$
(26)

3.2 Decode-and-forward relay transmission

For a decode-and-forward relay system, if the relay cannot correctly decode the signal from the source, then it will fall back to the direct transmission, due to the disadvantage of a fixed decode-and-forward relay system. Therefore, the outage probability can be expressed as follows:

$$\begin{array}{@{}rcl@{}} P_{out}&=&Pr\left( \frac{P_{0}|h_{0k}|^{2}}{N}\leq \gamma_{th}\right)Pr\left( \frac{P_{0}|h_{0c}|^{2}}{N}\leq \gamma_{th}\right)\\ &&+Pr\left( \frac{P_{0}|h_{0k}|^{2}}{N}> \gamma_{th}\right)\\ &&\quad Pr\left( \frac{P_{0}|h_{0c}|^{2}}{N}+\frac{P_{k}|h_{kc}|^{2}}{N}\leq \gamma_{th}\right). \end{array} $$
(27)

Consequently, the outage probability (the objective function) in the DF relay mode is given as:

$$\begin{array}{@{}rcl@{}} P_{out}=\frac{(\gamma_{th}N)^{2}}{|H_{0c}|^{2}|H_{0k}|^{2}}(\frac{1}{(P_{0})^{2}}+ \frac{|H_{0k}|^{2}}{2P_{0}P_{k}|H_{kc}|^{2}}). \end{array} $$
(28)

Similarly, the cognitive user S U 0 and the relay node S U k shall transmit at the maximum power in the condition of peak interference constraint. From Eqs. 17 and 18, the optimal solution can be written as:

$$\begin{array}{@{}rcl@{}} P^{\ast}_{0}=\frac{I_{p}}{|h_{0p}|^{2}},\ P^{\ast}_{k}=\frac{I_{p}}{|h_{kp}|^{2}}. \end{array} $$
(29)

Subject to the average interference constraint, the resulting optimization problem can be posed as:

$$\begin{array}{@{}rcl@{}} \min \ P_{out} \end{array} $$
$$\begin{array}{@{}rcl@{}} \text{subject \ to : the\ constraint } (19) \end{array} $$
(30)

Consequently, we have the Lagrangian function:

$$\begin{array}{@{}rcl@{}} L(P_{0},P_{k},\lambda_{2})&=&-\frac{(\gamma_{th}N)^{2}}{|H_{0c}|^{2}|H_{0k}|^{2}}\left( \frac{1}{(P_{0})^{2}}+ \frac{|H_{0k}|^{2}}{2P_{0}P_{k}|H_{kc}|^{2}}\right)\\ &&-\lambda_{2}(P_{0}|h_{0p}|^{2}+P_{k}|h_{kp}|^{2}-2I_{p}), \end{array} $$
(31)

where λ 2 is the Lagrangian multiplier. According to

$$\begin{array}{@{}rcl@{}} \frac{\partial L}{\partial P_{0}}=0 \ , \ \frac{\partial L}{\partial P_{k}}=0, \end{array} $$
(32)

the optimum power allocation can be shown as:

$$\begin{array}{@{}rcl@{}} P^{\ast}_{0}=\frac{-b\pm\sqrt{b^{2}-4ac}}{2a}, \end{array} $$
$$\begin{array}{@{}rcl@{}} P^{\ast}_{k}=\frac{2I_{p}-P^{\ast}_{0}|h_{0p}|^{2}}{|h_{kp}|^{2}}, \end{array} $$
(33)

where

$$\begin{array}{@{}rcl@{}} a=2|H_{0k}|^{2}|h_{0p}|^{2}|h_{kp}|^{2}-4|H_{kc}|^{2}|h_{0p}|^{4}, \end{array} $$
$$\begin{array}{@{}rcl@{}} b=(16|H_{kc}|^{2}|h_{0p}|^{2}-2|H_{0k}|^{2}|h_{kp}|^{2})I_{p}, \end{array} $$
$$\begin{array}{@{}rcl@{}} c=-16|H_{kc}|^{2}{I^{2}_{p}}. \end{array} $$
(34)

3.3 Hybrid decode-amplify-forward relay transmission

The idea of hybrid of the AF and DF protocol (HDAF) was introduced in [16]. In this section, we consider the relay node to use the HDAF protocol that converts the AF and DF scheme not successful decoding. The HDAF relay scheme combines the advantages of both the AF and DF relaying scheme. The HDAF operates in the DF mode if the relay node can decode the signal correctly. Otherwise, the HDAF will operate in the AF mode. The outage probability of the HDAF system can be written as:

$$\begin{array}{@{}rcl@{}} P_{out}&=&Pr\left( \frac{P_{0}|h_{0c}|^{2}}{N}<\gamma_{th}\right)\\&& \times\left[Pr\left( \frac{P_{0}|h_{0k}|^{2}}{N}<\gamma_{th}\right) Pr\left( \frac{P_{0}|h_{0c}|^{2}}{N}\right.\right.\\ &&\qquad\left.+f\left( \frac{P_{0}|h_{0k}|^{2}}{N},\frac{P_{k}|h_{kc}|^{2}}{N}\right)<\gamma_{th}\right)\\ &&\qquad+Pr\left( \frac{P_{0}|h_{0k}|^{2}}{N}>\gamma_{th}\right)Pr\left( \frac{P_{0}|h_{0c}|^{2}}{N}\right.\\ &&\left.\left.\qquad+\frac{P_{k}|h_{kc}|^{2}}{N}<\gamma_{th}\right)\right]. \end{array} $$

The outage probability expression for the HDAF relaying strategy can be calculated as:

$$\begin{array}{@{}rcl@{}} P_{out}=\frac{(\gamma_{th}N)^{3}}{2(P_{0})^{2}|H_{0c}|^{4}|H_{0k}|^{2}}\left( \frac{1}{P_{0}}+ \frac{|H_{0k}|^{2}}{P_{k}|H_{kc}|^{2}}\right). \end{array} $$
(35)

Similarly, the cognitive user S U 0 and the relay node S U k shall transmit at the maximum power in the condition of peak interference constraint. Subject to the peak interference constraint (17) and (18), the optimal solution can be written as:

$$\begin{array}{@{}rcl@{}} P^{\ast}_{0}=\frac{I_{p}}{|h_{0p}|^{2}},\ P^{\ast}_{k}=\frac{I_{p}}{|h_{kp}|^{2}}. \end{array} $$
(36)

For the average interference constraint, the power allocation problem to minimize the outage probability can be mathematically formulated as:

$$\begin{array}{@{}rcl@{}} \min \ P_{out} \end{array} $$
$$\begin{array}{@{}rcl@{}} \text{ subject \ to : the\ constraint }\ (19) \end{array} $$
(37)

Using the Lagrangian multiplier method, the modified objective function can be written as:

$$\begin{array}{@{}rcl@{}} L(P_{0},P_{k},\lambda_{3})&=&-\frac{(\gamma_{th}N)^{3}}{2(P_{0})^{2}|H_{0c}|^{4}|H_{0k}|^{2}}\left( \frac{1}{P_{0}}+ \frac{|H_{0k}|^{2}}{P_{k}|H_{kc}|^{2}}\right)\\ &&-\lambda_{3}(P_{0}|h_{0p}|^{2}+P_{k}|h_{kp}|^{2}-2I_{p}). \end{array} $$
(38)

Here, λ 3 serves as the Lagrangian multiplier related to the average interference constraint. According to

$$\begin{array}{@{}rcl@{}} \frac{\partial L}{\partial P_{0}}=0 \ , \ \frac{\partial L}{\partial P_{k}}=0, \end{array} $$
(39)

and by solving the set of equations in Eq. 39 for the HDAF relaying, it can be shown that the optimal values for P 0 and P k can be obtained as:

$$\begin{array}{@{}rcl@{}} P^{\ast}_{0}=\frac{-b\pm\sqrt{b^{2}-4ac}}{2a}, \end{array} $$
$$\begin{array}{@{}rcl@{}} P^{\ast}_{k}=\frac{2I_{p}-P^{\ast}_{0}|h_{0p}|^{2}}{|h_{kp}|^{2}}, \end{array} $$
(40)

where

$$\begin{array}{@{}rcl@{}} a=3|H_{0k}|^{2}|h_{0p}|^{2}|h_{kp}|^{2}-3|H_{kc}|^{2}|h_{0p}|^{4}, \end{array} $$
$$\begin{array}{@{}rcl@{}} b=(12|H_{kc}|^{2}|h_{0p}|^{2}-4|H_{0k}|^{2}|h_{kp}|^{2})I_{p}, \end{array} $$
$$\begin{array}{@{}rcl@{}} c=-12|H_{kc}|^{2}{I^{2}_{p}}. \end{array} $$
(41)

For each of the relay mode, the relay selection is to find the relay node that minimizes the outage probability. Specifically, the relay selection follows the following steps:

  1. 1)

    For the AF mode, we must solve the problems outlined in Eqs. 21 and 22 to obtain the values of \(P^{\ast }_{0}\) and \(P^{\ast }_{k}\) under both the peak and average interference constraints. For the DF mode, we must solve the problems outlined in Eqs. 29 and 30 to obtain the optimal power. For the HDAF mode, we must solve the problems outlined in Eqs. 36 and 37 to obtain the optimal power.

  2. 2)

    Substitute the values of the optimal power (\(P^{\ast }_{0}\) and \(P^{\ast }_{k}\)) obtained from Step 1 into Eqs. 20, 28, and 35 to calculate the outage probability \(P^{\ast }_{out}\) for various relay modes.

  3. 3)

    Among all of the possible relay nodes, we choose the optimal relay node k according to the following criterion:

    $$\begin{array}{@{}rcl@{}} k^{\ast}=\arg \min P^{k^{\ast}}_{out}. \end{array} $$
    (42)

The corresponding outage probability is represented by \(P^{k^{\ast }}_{out^{\ast }}\).

4 Simulation results

In this section, we show the performance of the proposed power allocation algorithm in the cognitive relay network under both the peak and average interference constraints. In particular, we analyze the outage probabilities in three different relay modes. To verify the theoretical foundations, we simulate all of the modes in Matlab. The simulation topology is shown in Fig. 2, where the AP, cognitive user S U 0, and the PU receiving terminal are located at coordinates (0, 0), (D, 0), and (D, D/2), respectively. The channel path loss factor is set as α = 3. The PU’s interference noise (INR) is used to indicate the interference constraint of secondary users.

Fig. 2
figure 2

Simulation topology

First, we consider a scenario as shown in Fig. 2. Two potential cognitive radio relays S U 1 and S U 2 are located at (D/2, D/4) and (D/2, -D/4), respectively. Figure 3 shows the outage performance in the AF mode under both of the peak and the average interference constraints. It can be seen that the outage probability with S U 2 is lower than that of the S U 1. This is because S U 2 is far away from the PU so that more transmitting power is utilized. It is noteworthy to mention that, without the interference constraint, the S U 1 and S U 2 will provide the same cooperative gain because of the equal distances to the S U 0 and AP. Compared with the peak interference constraint, the average interference constraint has lower outage probability, due to the power adaptation between the transmitter and relay. Similarly, Figs. 4 and 5 show the performance in the DF and HDAF modes under both of the peak and average interference constraints.

Fig. 3
figure 3

The AF performance comparison based cooperation link

Fig. 4
figure 4

The DF performance comparison based cooperation link

Fig. 5
figure 5

The HDAF performance comparison based cooperation link

To better compare the performance of the three different modes (AF, DF and HDAF ), Tables 1 and 2 show the outage performance under the peak interference constraint and the average interference constraint, respectively. We can observe that the HDAF mode has a lower outage probability and the DF has a higher outage probability when compared with the AF mode. On the other hand, the outage probability among the three relay schemes will decrease when the interference temperature becomes larger at the PU.

Table 1 Peak interference constraint
Table 2 Average interference constraint

Next, we study the effect of the number of relays on the outage performance. We randomly deployed K cognitive relays within a rectangular area ((0, -D/4), (D, -D/4), (0, D/4), and (D, -D/4)), as shown in Fig. 2. We set the PU’s interference temperature as 0 dB. Given the number of potential relay, the minimal outage probability is investigated, averaged over various relay distributions and fading channel realizations.

For all of the three relay modes, Fig. 6 shows the average minimal outage probability versus the number of cognitive relays under peak and average interference constraints. As expected, the outage probability decreases with the number of relays. Similar to the single relay case, we can observe that the HDAF mode and the DF mode have the best and the worst performance, respectively.

Fig. 6
figure 6

The outage probability versus the number of potential cognitive radio relays

5 Conclusions

In this paper, we studied the outage performance of three cooperative relay modes, namely: AF, DF, HDAF. The closed form of the outage probability was derived. If the quality of service was very rigid, it was beneficial that the cooperation was obtained at the cognitive user from the surrounding cognitive users to decrease the outage probability. The best relay when selected, can not only decrease the interference to the primary user but also improve the performance of the cognitive user. The simulation results show that the HDAF cooperative relay mode achieve better performance than the AF and DF cooperative relay mode. Moreover, the outage probability decreased as the number of potential relays increased. As a future work, we will consider the extension of the protocols to multi-node scenarios where multiple terminals communicate with multiple partner terminals.