1 Introduction

Non-orthogonal multiple access (NOMA) is one of the promising candidates for enhancing power spectrum efficiency of 5 G cellular networks [1, 2]. Orthogonal multiple access (OMA) methods are designed to eliminate interference between users. However, these methods have low spectral efficiency due to exclusive assignment of resources [3]. On the other hand, by ever increasing number of users, the utilization of new methods with higher spectral efficiency is vital in the design of next generation of networks. In NOMA systems each resource will be assigned to at least two users. Although sharing resources, improves spectral efficiency and throughput [4, 5], there are some challenges that should be addressed too [6]. Due to the interference between the users that are sharing resources, receivers must have the ability to detect symbols and this increases the complexity of receivers.

Two fundamental elements for implementing a NOMA based network are superposition coding at the transmitter and successive interference cancellation (SIC) at destinations [7]. By superposition coding, the symbols of users are combined with specific power scales at the transmitter to be sent to the users [8]. Although system performance can be enhanced by applying channel coding [9]. Successive interference cancellation is a technique for detection of symbol of users that are linearly combined, e.g. through superposition coding, and its performance highly depends on power allocation [10, 11].

In NOMA based networks, usually two users are paired to share a common subcarrier for data transmission between base station (BS) and destinations [12]. More power is allocated to the user with weaker channel condition, and this user can decode its own data by a simple matched filtering. In contrast, the user with stronger channel condition will be allocated less power and should apply SIC to decode its own data [13].

Cooperative communication is a well-known technique to achieve spatial diversity to combat fading effects [14] and also improving the coverage in wireless systems [15]. In cooperative networks, several relays assist the transmitter for data transmission. Two protocols that are widely used in relaying systems, include amplify and forward (AF) and decode and forward (DF) relaying [16]. In AF method, relay retransmits the received signal, while in DF relaying relay decodes the received symbols, and then retransmits them [17]. System performance depends on the number of relays and system structure, but it is generally proved that cooperative communication increases capacity and throughput. Using relays in NOMA systems, can improve the spectral efficiency and bit error rate (BER) performance of communication systems simultaneously [18]. In [18], a cooperative non-orthogonal multiple access (NOMA) system is proposed, where one near user is employed as decode-and-forward (DF) relaying switching between full-duplex (FD) and half-duplex (HD) mode to help a far user. Performance metrics are outage probability, ergodic rate and energy efficiency. In [19] the source transmits two symbols using the superposition code, and the relay decodes and retransmits the symbol of the user with lower allocated power by employing the successive interference cancellation (SIC). At the destination, two symbols from both the direct signal and the forwarded signal are decoded by using the maximum-ratio combining and the ergodic sum rate and the outage performance of the system are investigated.

A model of a cooperative network with K users is considered in [20]. There is no dedicated relay, and users assist each other for data transmission. Based on SIC technique, each user should decode symbol of users with more power; hence, it can act as a relay and send its estimation to stronger users. The outage probability and ergodic sum rate of this model is calculated based on SINR of each user. The main drawback of this model is that \((K-1)\) time slots are required to send one symbol. As a consequence, the delay of the system will be increased by the number of users. One of the main benefits of cooperative networks is improving coverage [15].

Power allocation and user pairing have a significant effect on the performance of NOMA based networks. In [21], an algorithm is suggested for maximizing power efficiency, by assuming perfect channel state information (CSI). In [22, 23], the impact of user paring on NOMA based networks are investigated and it is shown that the best performance corresponds to a scenario in which users with distinctive channel conditions are paired.

In this paper, the BER performance of a cooperative NOMA system is investgated. A base station communicates with two users by NOMA with the assistance of a DF relay. After combining the received signals from the direct and cooperative paths, users decode their symbols using SIC technique. The BER performance of proposed combining scheme is computed in closed form for BPSK modulation and can be genelalized to arbitrary M-ary constellations. Further, a Min–Max optimization is proposed for finding the optimum power allocation, and an approximate solution is derived. Besides, the diversity order of proposed scheme is analyzed and it is shown the the proposed method achieves full diversity order for both users if the proper power allocation condition for users are met. Performance of proposed scheme is investigated through simulations which corroborates the analytical results. The main contribution of this paper is the investigation of BER metric and diversity order for NOMA based cooperative systems while other related works consider metrics like energy efficiency, ergodic capacity and outage probability. Furthermore this paper investigates the diversity order of proposed method and derives sufficient conditions under which full diversity order can be achieved.

The rest of this paper is organized as follows. In Sect. 2, the model of system under investigation is described. In Sect. 3, the proposed receiver is introduced, and its BER performance for BPSK modulation is analyzed in Sect. 4. Power allocation problem is formulated in Sect. 5. In Sect. 6, diversity order of the proposed receiver is evaluated. In Sect. 7 results are extended to any arbitrary constellation. Simulation and numerical results are presented in Sect. 8. Finally, conclusions are drawn in Sect. 9.

2 System model

In this paper, downlink of a NOMA based network with two users is considered where a relay assists the BS for data transmission. With reference to Fig. 1, our model consists of a source (S) and a DF relay (R) and two destinations (\(D_1, D_2\)).

Fig. 1
figure 1

Block diagram of a cooperative network with two users

By employing superposition coding at the BS, symbol of users are combined with specific scales \(\alpha \) and \(\beta \) to be transmitted. By considering time division duplex (TDD) mode, data transmission consists of two time slots. In the first time slot, S broadcast coded signal (\(\alpha S_{1}+\beta S_{2}\)) to relay and both destinations. Received symbols at R, \(D_1\) and \(D_2\) can be written as,

$$\begin{aligned} y_{SR}= & {} h_{SR}(\alpha S_{1}+\beta S_{2})+n_{SR} \end{aligned}$$
(1)
$$\begin{aligned} y_{SD_1}= & {} h_{SD_1}(\alpha S_{1}+\beta S_{2})+n_{SD_1} \end{aligned}$$
(2)
$$\begin{aligned} y_{SD_2}= & {} h_{SD_2}(\alpha S_{1}+\beta S_{2})+n_{SD_2} \end{aligned}$$
(3)

where information symbols denoted by \(S_1, S_2 \in {\mathcal {A}}_M\) are drawn form any arbitrary M-ary constellation \({\mathcal {A}}_M\) and \(n_{SR}\), \(n_{SD_i}\), \(n_{RD_i}\) (i=1,2) are additive noise of corresponding channels that are assumed to be zero mean complex Gaussian random variables with variance \(N_{0}\). In the second time slot, relay exploits successive interference cancellation (SIC) in order to decode \(S_{1}\) and \(S_{2}\) from the received signal. Then, using superposition codes, relay combines decoded symbols and transmits it to \(D_1\) and \(D_2\). The received signals at each destination can be expressed as

$$\begin{aligned} y_{RD_1}= & {} h_{RD_1}({\hat{\alpha }} {\hat{S}}_{1}+{\hat{\beta }} {\hat{S}}_{2})+n_{RD_1} \end{aligned}$$
(4)
$$\begin{aligned} y_{RD_2}= & {} h_{RD_2}({\hat{\alpha }} {\hat{S}}_{1}+{\hat{\beta }} {\hat{S}}_{2})+n_{RD_2} \end{aligned}$$
(5)

where \({\hat{S}}_{1}\) and \({\hat{S}}_{2}\) denote detected symbols corresponding to \(S_1\) and \(S_2\) at relay, and \({\hat{\alpha }}\) and \({\hat{\beta }}\) are power allocation scales at relay. The fading coefficient of each link is modeled as zero mean complex Gaussian random variable:

$$\begin{aligned} \begin{aligned}&h_{SR} \sim {{\mathcal {C}}}{{\mathcal {N}}}(0,\sigma _{SR}^2)~~~~\sigma _{SR}^2=E\{|h_{SR}|^2\}\\&h_{SD_i} \sim {{\mathcal {C}}}{{\mathcal {N}}}(0,\sigma _{SD_i}^2)~~~~\sigma _{SD_i}^2=E\{|h_{SD_i}|^2\}~~~~i=1,2\\&h_{RD_i} \sim {{\mathcal {C}}}{{\mathcal {N}}}(0,\sigma _{RD_i}^2)~~~~\sigma _{RD_i}^2=E\{|h_{RD_i}|^2\}~~~~i=1,2 \end{aligned} \end{aligned}$$

Instantaneous signal-to-noise-ratio (SNR) at each link is defined as,

$$\begin{aligned} \begin{aligned}&\gamma _{SR}=|h_{SR}|^2 \bar{\gamma }\\&\gamma _{SD_1}=|h_{SD_1}|^2 \bar{\gamma } ~,~\gamma _{RD_1}=|h_{RD_1}|^2 \bar{\gamma }\\&\gamma _{SD_2}=|h_{SD_2}|^2 \bar{\gamma }~,~\gamma _{RD_2}=|h_{RD_2}|^2 \bar{\gamma } \end{aligned} \end{aligned}$$
(6)

where \(\bar{\gamma }=\frac{P_{s}(\alpha ^2+\beta ^2)}{N_{0}}\) and \(P_s\) is the transmitted power at source. We also assume combining coefficients \(\alpha \) and \(\beta \) meet the constraint \( \alpha ^2+\beta ^2=1\) and \(P_s=E\{ {S_1}^2\}=E\{ {S_2}^2\} \).

In the rest of this paper, we assume similar power allocation at BS and relay, i.e., (\({\hat{\alpha }}=\alpha \) , \({\hat{\beta }}=\beta \)), unless mentioned otherwise. In Sect. 8, we will show that this assumption will not affect the performance, significantly.

As mentioned before, the most important point in separating symbol of users is power allocation. Based on SIC features, for the feasibility of detection, the user with stronger channel condition should be allocated less power than the user with weaker channel condition. Throughout this paper, we will call the user with more allocated power (user with weaker channel coefficient) as "strong user" and the user with less allocated power (user with stronger channel coefficient) by "weak user".

3 Proposed receiver

Consider a combing scheme, where received signals by direct and relaying phase are linearly combined by \(w_{SD_1}\) and \(w_{RD_1}\) for user 1, and \(w_{SD_2}\) and \(w_{RD_2}\) for user 2. The combiner output for each user can be written as,

$$\begin{aligned} y_{D_1}= & {} w_{SD_1}y_{SD_1}+w_{RD_1}y_{RD_1} \end{aligned}$$
(7)
$$\begin{aligned} y_{D_2}= & {} w_{SD_2}y_{SD_2}+w_{RD_2}y_{RD_2} \end{aligned}$$
(8)

Proposed weights for combining are [24],

$$\begin{aligned}{} & {} w_{SD_1}= h_{SD_1}^*, ~~w_{RD_1}=\frac{\gamma _{min_1}}{\gamma _{RD_1}} h_{RD_1}^*\end{aligned}$$
(9)
$$\begin{aligned}{} & {} \quad w_{SD_2}= h_{SD_2}^*,~~w_{RD_2}=\frac{\gamma _{min_2}}{\gamma _{RD_2}} h_{RD_2}^*\end{aligned}$$
(10)

where \(\gamma _{min_1}\) and \(\gamma _{min_2}\) are defined as

$$\begin{aligned} \gamma _{min_i}=min\{\ \gamma _{SR},\gamma _{RD_i}\} \quad (i=1,2) \end{aligned}$$
(11)

By adjusting the weights using (9) and (10), signals will be combined constructively and moreover, the contribution of the relaying phase will be adjusted based on the SNR of S-R-D link. Consider a scenario in which the SNR of S-R link is low; consequently, the probability of erroneous detection at relay will increase. In this case, weight of the relaying phase should be decreased. In contrast, If the SNR of S-R link is acceptable, weight of the relaying phase increases.

Using SIC, the strong user will decode its symbols through a simple match filtering, whereas the weak user should first decode the symbols of the strong user and after substracting it from the received signal, then decode its own symbols. For simplicity of presentation, we assume that user 1 is the strong user and user 2 is the weak user (user 1 has more power than user 2).

Relay should also decode \(S_{1}\) and \(S_{2}\) through SIC. Decision rules at relay are,

$$\begin{aligned} \begin{aligned}&{\hat{S}}_{1}= \underset{S_{1}\in {\mathcal {A}}_M}{{\text {arg~min}}}\{| y_{SR}-h_{SR}~\alpha ~S_{1}|^2\\&\quad \hat{S_2}=\underset{S_{2}\in {\mathcal {A}}_M}{{\text {arg~min}}}\{| y_{SR}-h_{SR}~(\alpha ~{\hat{S}}_{1}+\beta ~S_{2})|^2\} \end{aligned} \end{aligned}$$
(12)

Users receive signals from direct and cooperative paths, and combine them using (7) and (8). The strong user does not need to perform SIC, and the decision rule for this user is,

$$\begin{aligned} \begin{aligned} \hat{S_1}= \underset{S_{1}\in {\mathcal {A}}_M}{{\text {arg~min}}}\{| y_{D_1} -(w_{SD_1}h_{SD_1}+w_{RD_1}h_{RD1})\alpha S_{1}|^2\} \end{aligned}\nonumber \\ \end{aligned}$$
(13)

The weak user first decodes symbol of strong user by the decision rule,

$$\begin{aligned} \begin{aligned} {\hat{S}}_{1,2}{=} \underset{S_{1}\in {\mathcal {A}}_M}{{\text {argmin}}}\{| y_{D_2} {-}(w_{SD_2}h_{SD_2}{+}w_{RD_2}h_{RD_2})\alpha S_{1}|^2\} \end{aligned} \end{aligned}$$
(14)

then using \({\hat{S}}_{1,2}\) of (14), the weak user decodes \(\hat{S_2}\) as,

$$\begin{aligned} \begin{aligned}&\hat{S_2}=\underset{S_{2}\in {\mathcal {A}}_M}{{\text {argmin}}}\{| y_{D_2} -(w_{SD_2}h_{SD_2}+w_{RD_2}h_{RD_2})\\&\times (\alpha {\hat{S}}_{1,2}+\beta S_{2})|^2\} \end{aligned} \end{aligned}$$
(15)

4 BER performance for BPSK modulation

Although the proposed receiver can be used for any constellation, in the first step, we analyze the BER performance of BPSK signaling. The results will be extended to any M-ary constellation in Sect. 7.

4.1 Error probability at relay

Relay decodes \(S_{1},S_2 \in \{\pm \sqrt{P_{s}}\}\) from the received signal \(y_{SR}\) given in (1) using (12). First, the symbol of the strong user and then after subtraction, the symbol of the weak user will be detected. By defining \(p_{strong}\)= \(max\{ \alpha ,\beta \}\) and \(p_{weak}\)= \(min\{ \alpha ,\beta \}\), the error probability of the strong user at relay can be expressed as,

$$\begin{aligned}{} & {} Pe_{Strong}=0.5~Q\left( \frac{\sqrt{P_{s}}|h_{SR}|(p_{Strong}-p_{Weak})}{\sqrt{\frac{N_0}{2}}}\right) \nonumber \\{} & {} \quad +0.5~Q\left( \frac{\sqrt{P_{s}}|h_{SR}|(p_{Strong}+p_{Weak})}{\sqrt{\frac{N_0}{2}}}\right) \end{aligned}$$
(16)

The weak user should subtract \(\alpha \hat{S_1} h_{SR}\) from the received signal. If \(S_{1}\) is detected correctly, the interference will be eliminated after subtraction, whereas in BPSK modulation the interference will be doubled in case of erroneous detection. Thus, the error probability of the weak user can be calculated as,

$$\begin{aligned}{} & {} Pe_{Weak}=(1-Pe_{Strong})Q\left( \frac{\sqrt{P_s}|h_{SR}|p_{Weak}}{\sqrt{\frac{N_0}{2}}}\right) \nonumber \\{} & {} \quad +Pe_{Strong}\bigg \{ 0.5~Q\left( \frac{\sqrt{P_s}|h_{SR}|(p_{Weak}-2~p_{Strong})}{\sqrt{\frac{N_0}{2}}}\right) \nonumber \\{} & {} \quad +0.5~Q\left( \frac{\sqrt{P_s}|h_{SR}|(p_{Weak}+2~p_{Strong})}{\sqrt{\frac{N_0}{2}}}\right) \bigg \}\nonumber \\ \end{aligned}$$
(17)

4.2 Error probability of strong user

Using (2), (4) and (7), the combiner output for the strong user (user 1) can be written as

$$\begin{aligned} \begin{aligned}&y_{D_1}=w_{SD_1}[h_{SD_1}(\alpha S_{1}+\beta S_{2})+n_{SD_1}] \\&+w_{RD_1}[h_{RD_1}(\alpha {\hat{S}}_{1}+\beta {\hat{S}}_{2})+n_{RD_1}] \end{aligned} \end{aligned}$$
(18)

With BPSK modulation the detected symbols of each user can only take two values. For instance, \({\hat{S}}_{1}\)=\(\pm S_{1}\) for user 1. Therefore, the combiner output will assume four possible forms as presented in (19), where \(N_{1}=w_{SD_1}n_{SD_1}+w_{RD_1}n_{RD_1}\).

$$\begin{aligned} \begin{aligned} y_{D_1}=\left\{ \begin{array}{ll} (w_{SD_1}h_{SD_1}+w_{RD_1}h_{RD_1})\alpha S_{1} +(w_{SD_1}h_{SD_1}+w_{RD_1}h_{RD_1})\beta S_{2}+N_{1} &{} ~~~\text {if } {\hat{S}}_{1}=S_{1}, {\hat{S}}_{2}=S_{2} \\ (w_{SD_1}h_{SD_1}+w_{RD_1}h_{RD_1})\alpha S_{1} +(w_{SD_1}h_{SD_1}-w_{RD_1}h_{RD_1})\beta S_{2}+N_{1}&{} ~~~\text {if } {\hat{S}}_{1}=S_{1}, {\hat{S}}_{2}=-S_{2} \\ (w_{SD_1}h_{SD_1}-w_{RD_1}h_{RD_1})\alpha S_{1} +(w_{SD_1}h_{SD_1}+w_{RD_1}h_{RD_1})\beta S_{2}+N_{1}&{}~~~ \text {if } {\hat{S}}_{1}=-S_{1}, {\hat{S}}_{2}=S_{2} \\ (w_{SD_1}h_{SD_1}-w_{RD_1}h_{RD_1})\alpha S_{1} +(w_{SD_1}h_{SD_1}-w_{RD_1}h_{RD_1})\beta S_{2}+N_{1}&{}~~~ \text {if } {\hat{S}}_{1}=-S_{1}, {\hat{S}}_{2}=-S_{2} \end{array} \right. \end{aligned} \end{aligned}$$
(19)
$$\begin{aligned}{} & {} Pe_{1}=(1-B)(1-A)\left[ 0.5Q\left( \frac{|H_{1}|\alpha -|H_{1}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}\right) \right] \nonumber \\{} & {} \quad +\left[ 0.5Q\left( \frac{|H_{1}|\alpha +|H_{1}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}\right) \right] \nonumber \\{} & {} \quad +B(1-A)\left[ 0.5Q\left( \frac{|H_{1}|\alpha -|H_{2}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}\right) \right] \nonumber \\{} & {} \quad +\left[ 0.5Q\left( \frac{|H_{1}|\alpha +|H_{2}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}\right) \right] \nonumber \\{} & {} \quad +(1-C)A\left[ 0.5Q\left( \frac{|H_{2}|\alpha -|H_{1}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}\right) \right] \nonumber \\{} & {} \quad +\left[ 0.5Q\left( \frac{|H_{2}|\alpha +|H_{1}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}\right) \right] \nonumber \\{} & {} \quad + CA\left[ 0.5Q\left( \frac{|H_{2}|\alpha -|H_{2}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}\right) \right] \nonumber \\{} & {} \quad +\left[ 0.5Q\left( \frac{|H_{2}|\alpha +|H_{2}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}\right) x\right] \end{aligned}$$
(20)

Using (19), the error probability of the strong user can be computed as (20), where \(H_{1}\), \(H_{2}\) and \(\sigma _{N_{1}}^2\) are defined as

$$\begin{aligned} \begin{aligned}&H_{1}=\sqrt{P_{s}} (w_{RD_1}h_{RD_1}+w_{SD_1}h_{SD1} )\\&H_{2}=\sqrt{P_{s}} (-w_{RD_1}h_{RD_1}+w_{SD_1}h_{SD1})\\&\sigma _{N_{1}}^2=\big (|w_{RD_1}|^2+|w_{SD_1}|^2 \big )N_{0} \end{aligned} \end{aligned}$$
(21)

In this equation A (probability of wrong detection of strong user at relay), B (probability of wrong detection of weak user conditioned on correct detection of strong user at relay) and C (probability of wrong detection of weak user conditioned on wrong detection of strong user at relay)are defined as follows:

$$\begin{aligned}{} & {} A=0.5~Q\left( \frac{|h_{SR}|(\alpha -\beta )}{\sqrt{\frac{{\sigma _{SR}}^2}{2}}}\right) +0.5~Q\left( \frac{|h_{SR}|(\alpha +\beta )}{\sqrt{\frac{{\sigma _{SR}}^2}{2}}}\right) \nonumber \\{} & {} \quad B=Q\left( \frac{|h_{SR}|\beta }{\sqrt{\frac{{\sigma _{SR}}^2}{2}}}\right) \nonumber \\{} & {} \quad C=\left[ 0.5~Q\left( \frac{(\beta -2~\alpha )|h_{SR}|}{\sqrt{\frac{{\sigma _{SR}}^2}{2}}}\right) +0.5~Q\left( \frac{(\beta +2~\alpha )|h_{SR}|}{\sqrt{\frac{{\sigma _{SR}}^2}{2}}}\right) \right] \nonumber \\ \end{aligned}$$
(22)

4.3 Error probability of weak user

From (3), (5) and (8), the combiner output for the weak user (user 2) can be written as

$$\begin{aligned} \begin{aligned} y_{D_2}=w_{SD_2}\left[ h_{SD_2}(\alpha S_{1}+\beta S_{2})+n_{SD_2}\right] \\ +w_{RD_2}\left[ h_{RD_2}(\alpha {\hat{S}}_{1}+\beta {\hat{S}}_{2})+n_{RD_2}\right] \end{aligned} \end{aligned}$$
(23)

Similarly, four scenarios are possible regarding the detection at relay. Considering (14) and (15), the error probability of the weak user can be derived as (31) where \(P{e_{21}}\), \(P{e_{22}}\), \(P{e_{23}}\) and \(P{e_{24}}\) are the error probability of decoding symbols of the strong user by weak user in each scenario and are defined as

$$\begin{aligned}{} & {} P{e_{21}}=0.5Q\left( \frac{|G_{1}|\alpha -|G_{1}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \nonumber \\{} & {} \quad +0.5Q\left( \frac{|G_{1}|\alpha +|G_{1}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \nonumber \\{} & {} \quad P{e_{22}}=0.5Q\left( \frac{|G_{1}|\alpha -|G_{2}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \nonumber \\{} & {} +0.5Q\left( \frac{|G_{1}|\alpha +|G_{2}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \nonumber \\{} & {} \quad P{e_{23}}=0.5Q\left( \frac{|G_{2}|\alpha -|G_{1}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \nonumber \\{} & {} +0.5Q\left( \frac{|G_{2}|\alpha +|G_{1}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \nonumber \\{} & {} \quad P{e_{24}}=0.5Q\left( \frac{|G_{2}|\alpha -|G_{2}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \nonumber \\{} & {} +0.5Q\left( \frac{|G_{2}|\alpha +|G_{2}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \end{aligned}$$
(24)

where \(G_{1}\), \(G_{2}\) and \(\sigma _{N_2}^2\) are defined as,

$$\begin{aligned}{} & {} G_{1}=\sqrt{P_s}(w_{RD_2}h_{RD_2}+w_{SD_2}h_{SD2})\nonumber \\{} & {} \quad G_{2}=\sqrt{P_s} (-w_{RD_2}h_{RD_2}+w_{SD_2}h_{SD2})\nonumber \\{} & {} \quad \sigma _{N_{2}}^2=\big (|w_{RD_2}|^2+|w_{SD_2}|^2\big )N_{0} \end{aligned}$$
(25)

5 Optimum power allocation

In NOMA, users will be separated by power and as a consequence, the system performance strongly depends on power allocation adjustment. Generally, more power should be allocated to the user with weaker channel condition, but for finding the optimum power allocation an optimization problem should be defined. Different type of objectives can be considered for the optimization problem such as outage probability and ergodic sum rate. The goal of this paper is proposing a receiver with acceptable BER performance for both users. In this regard, we define an optimization problem in order to minimize the maximum error probability of users subject to the condition \(\alpha ^2+\beta ^2=1\).

The optimization problem can be presented as,

$$\begin{aligned} \begin{array}{ll} \min &{}\quad \max ~~ (Pe_{1},Pe_{2})\\ \quad s.t &{}\quad \alpha ^2+\beta ^2=1\\ &{}\qquad 0<\alpha , \beta <1 \end{array}. \end{aligned}$$
(26)

where \(Pe_{1}\) and \(Pe_{2}\) are defined as (20) and (31). Since the optimization problem is not convex, One possible approach for solving this problem is using an exhaustive search which may have high computational complexity. Another approach is finding an approximate solution for (26). We should note that the solution of (26) is the one of the points where \(Pe_{1}=Pe_{2}\).

If we only consider the dominant terms for\(Pe_{1}\) and \(Pe_{2}\), the approximate equation can be written as,

$$\begin{aligned} Q\left( \frac{|H_{1}|\alpha -|H_{1}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}x\right) =Q\left( \frac{|G_{1}\beta |}{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \end{aligned}$$
(27)

The summation can be bounded by using the inequality below,

$$\begin{aligned} Q(\sqrt{x})+Q(\sqrt{y})<Q\left( \sqrt{\min (x,y)}\right) \end{aligned}$$

Thus,

$$\begin{aligned} Q\left( \frac{|H_{1}|\alpha -|H_{1}|\beta }{\sqrt{\frac{{\sigma _{N_1}}^2}{2}}}\right) =Q\left( \frac{|G_{1}\beta |}{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\right) \end{aligned}$$
(28)

By substituting \(H_1\) with \(G_1\), we have

$$\begin{aligned}{} & {} Q(\sqrt{\frac{(\alpha -\beta )^2}{N_{0}/2} \left( |h_{SD_1}^2|+\frac{\gamma _{min_1}}{\gamma _{RD_1}}|h_{RD_1}|^2\right) })\nonumber \\{} & {} \quad =Q\left( \sqrt{\frac{\beta ^2}{N_{0}/2} (|h_{SD_2}^2|+\frac{\gamma _{min_2}}{\gamma _{RD_2}}|h_{RD_2}|^2)}\right) \end{aligned}$$
(29)

By solving this equation, a closed form solution for optimum power allocation problem can be calculated for BPSK modulation as

$$\begin{aligned} \begin{aligned}&\alpha =\sqrt{\frac{\eta }{1+\eta }}, ~~ \beta =\sqrt{\frac{1}{1+\eta }}\\&\eta \approx (\sqrt{\frac{\bar{\gamma }_{SD_2}+min\{\bar{\gamma }_{SR},\bar{\gamma }_{RD_2}\}}{\bar{\gamma }_{SD_1}+min\{\bar{\gamma }_{SR},\bar{\gamma }_{RD_1}\}}}+1)^2 \end{aligned} \end{aligned}$$
(30)

In the simulation section, we will show that the proposed approximation has a good performance very close to the optimal power allocation.

$$\begin{aligned} \begin{aligned} Pe_{2}&=(1-B)(1-A)\bigg [(1-P{e_{21}})Q\bigg (\frac{|G_{1}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\\&\quad +P{e_{21}}~[0.5Q\bigg (\frac{|G_{1}|\beta -2|G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\\&+0.5Q\bigg (\frac{|G_{1}|\beta +2|G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )]\bigg ]\\&\quad +B(1-A)\bigg [(1-P{e_{22}})Q\bigg (\frac{|G_{2}|\beta }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\\&\quad +P{e_{22}}~[0.5Q\bigg (\frac{|G_{2}|\beta -2|G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\\&\quad +0.5Q\bigg (\frac{|G_{2}|\beta +2|G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )]\bigg ]\\&\quad {+}(1-C)A\bigg [(1{-}P{e_{23}})\bigg [0.5~Q\bigg (\frac{|G_{1}|\beta {-}|G_{2}{-}G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\\&\quad +0.5~Q\bigg (\frac{|G_{1}|\beta +|G_{2}-G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\bigg ]\\&\quad +P{e_{23}}~[0.5Q\bigg (\frac{|G_{1}|\beta -|G_{2}+G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\\&\quad +0.5Q\bigg (\frac{|G_{1}|\beta +|G_{2}+G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )]\bigg ]\\&\quad +CA\bigg [(1-P{e_{24}})\bigg [0.5~Q\bigg (\frac{|G_{2}|\beta -|G_{2}-G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\\&\quad +0.5~Q\bigg (\frac{|G_{2}|\beta +|G_{2}-G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\bigg ]\\&\quad +P{e_{24}}~[0.5Q\bigg (\frac{|G_{2}|\beta -|G_{2}+G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )\\&\quad +0.5Q\bigg (\frac{|G_{2}|\beta +|G_{2}+G_{1}|\alpha }{\sqrt{\frac{{\sigma _{N_2}}^2}{2}}}\bigg )]\bigg ] \end{aligned}\nonumber \\ \end{aligned}$$
(31)

6 Diversity analysis of DF relaying

Diversity order is defined as the negative exponent of the average BER when SNR tends to infinity. As a matter of fact,

$$\begin{aligned} ~~~~~~~~~~~~~~P^b_{{\bar{\gamma }} \rightarrow \infty }\approx (G_c \bar{\gamma })^{-G_d} \end{aligned}$$
(32)

the diversity order is \(G_d\), and \(G_c\) denotes the coding gain. In this section, It will be proved that the proposed receiver can achieve diversity up to the order of two which means that it achieves full diversity. We should notice that the error probability of both users that demonstrated in (20) and (31) are instantaneous error probabilities. For evaluating the diversity order of proposed receiver, first the average BER of users will be calculated, and later the diversity orders will be assessed by bounding the average BER. Finally, by using proposition 1 (see “Appendix A” for proof) and proposition 2 (see “Appendix B” for proof) it will be proved that \(Pe_{1}\) and \(Pe_{2}\) achieve full diversity order.

Proposition 1: The expectation \(E\{ P_1\}\), where x is a positive constant and \(\gamma _{min}=min \{\gamma _{SR}, \gamma _{RD} \}\) achieves diversity order of two.

$$\begin{aligned} \begin{aligned} P_{1}=Q(\frac{\sqrt{2}x(\gamma _{SD}+\gamma _{min})}{\sqrt{\gamma _{SD}+\frac{\gamma _{min}^2}{\gamma _{RD}})}})~~~~~~ \end{aligned} \end{aligned}$$
(33)

See the proof in “Appendix A”.

Proposition 2: If \(16y^2(x+z^2)-(y+z)^4>0\), the expectation of (34) achieves diversity order of two, where x,y,z are positive constants and \(\gamma _{min}=min \{\gamma _{SR}, \gamma _{RD} \}\).

$$\begin{aligned} P_2=Q(\sqrt{2x\gamma _{SR}})Q\left( \frac{\sqrt{2}(y\gamma _{SD}-z\gamma _{min})}{\sqrt{\gamma _{SD}+\frac{\gamma _{min}^2}{\gamma _{RD}}}}\right) \end{aligned}$$
(34)

See the proof in “Appendix A”.

With reference to (20), the upper bound of \(Pe_{1}\) can be calculated by eliminating negative terms. All terms of the upper bound are in the form of proposition 1 and 2. Since we have proved that the average of error probabilities which are in form of proposition 1 and 2, achieve diversity order of two, it is clear that the strong user achieves full-diversity.

For the weak user, in the same manner of the strong user, first the upper bound will be calculated by removing negative elements. By taking the same steps, we can see that all elements of the upper bound of \(Pe_{2}\) are in the form of proposition 1 and 2, and based on these theorems, will achieve diversity order of two. Since all elements of \(Pe_{2}\) achieve diversity order of two, we can claim that the proposed receiver achieves full-diversity order for weak user too.

To use proposition 2 for diversity analysis, we should be confident that the condition is satisfied. Proposition 3 provides a lower bound for power allocation ratio that guarantees diversity order of proposed receiver.

Proposition 3: If \(k>2.5735\) (\(k=\frac{\alpha }{\beta }\)), the error probability of both users decays with order of two in high SNR regime.

See the proof in “Appendix A”.

By solving (26), the optimum power allocation will be drawn as \((\alpha ,\beta )=(0.94,0.3412)\). Since \(k>2.5735\), we can claim that the condition of property 2 is satisfied for all terms with optimum power allocation.

7 Performance bound for general constellations

We want to prove that the proposed receiver achieves full-diversity for any M-ary constellation \( {\mathcal {A}}_M \). First, we evaluate the union bound of a simple SIC receiver with two users.

Consider a system that consists of a BS and two users. If the BS exploits NOMA for data transmission, the received signal at destinations will be,

$$\begin{aligned} y_i=h_i (\alpha S_1+ \beta S_2) ~~~~~~ i=1,2 \end{aligned}$$
(35)

User with greater power (user 1 in our notation) decode its symbols as

$$\begin{aligned} \hat{S_1}=\underset{S_{1}\in {\mathcal {A}}_M }{{\text {argmin}}}\{ |y_{1}-h_{1}\alpha S_{1}|^2\} \end{aligned}$$
(36)

Assuming equiprobable symbols, the error probability for the first user can be bounded as (37), where \(d_{min}\) and \(d_{max}\) denote the minimum and maximum Euclidean distance between symbols in constellation \( {\mathcal {A}}_M \). At the worse case, the decoded symbol can be located at distance of \(d_{min}/2\) form the actual symbol in constellation.

$$\begin{aligned}{} & {} P_{e_{1}}=\frac{1}{M}\sum _{n=0}^{M-1} P_{e_{1}|S_{2}=S_{n}} \le P_{e_{1}|S_{2}=S_{N}}\nonumber \\{} & {} =P\bigg \{h_{1}(\alpha S_{1}+\beta S_{n})+n_{1}-h_{1}\alpha S_{1}\le \frac{d_{min}}{2}|h_{1}|\alpha \bigg \}\nonumber \\{} & {} =Q\left( \frac{\frac{d_{min}}{2}\alpha |h_{1}|-\beta |h_{1}||S_{n}|}{\sqrt{N_{0}/2}}\right) \nonumber \\{} & {} \le Q\left( \frac{\alpha |h_{1}|\frac{d_{min}}{2}-|h_{1}|\beta \frac{d_{max}}{2}}{\sqrt{\frac{N_{0}}{2}}}\right) \end{aligned}$$
(37)

where \(S_{N}\) denotes the nearest point in the constellation to \(S_{1}\). The weak user should apply SIC. Decision rule for this user is,

$$\begin{aligned} \begin{aligned}&{\hat{S}}_{2}= \underset{S_{2}\in S_{M}}{{\text {arg~min}}}\{| \hat{y_{2}}-h_{2}~\beta ~S_{2}|^2\}\\&\quad \hat{y_{2}}=y_{2}-h_{2}\alpha \hat{S_1}=h_{2}\alpha (S_{1}-\hat{S_1})+h_{2}\beta S_{2}+n_{2} \end{aligned} \end{aligned}$$
(38)

The error probability of weak user depends on the detection of \(\hat{S_1}\).

$$\begin{aligned} \begin{aligned}&P_{e_2}\le (1-P_{e_1}) \times P\left( n_{2}\ge \beta |h_{2}| \frac{d_{min}}{2}\right) \\&\quad + P_{e_1} \times P\left( h_{2} \alpha (S_{1}-\hat{S_1})+n_{2}\ge \beta |h_{2}| \frac{d_{min}}{2}\right) \end{aligned} \end{aligned}$$
(39)

Finally, the union bound of error probability for weak user can be bounded by,

$$\begin{aligned} \begin{aligned}&P_{e_{2}}\le P_{e_{1}}Q\left( \frac{\beta |h_{2}|\frac{d_{min}}{2}-h_{2}\alpha d_{max}}{\sqrt{\frac{N_{0}}{2}}}\right) \\&\quad +(1-P_{e_{1}}) Q(\frac{\beta |h_{2}|\frac{d_{min}}{2}}{\sqrt{\frac{N_{0}}{2}}}) \le Q\left( \frac{\beta |h_{2}| d_{min}}{\sqrt{2 N_{0}}}\right) \\&\quad +Q\left( \frac{\beta d_{min} |h_{2}|-2|h_{2}|\alpha d_{max}}{\sqrt{2 N_{0}}}\right) \\&\quad \times Q\left( \frac{\alpha d_{min} |h_{2}|-|h_{2}|\beta d_{max}}{\sqrt{2 N_{0}}}\right) \end{aligned} \end{aligned}$$
(40)

Using proposition 1 and proposition 2, the union bound of error probability decays with order of two when SNR tends to infinity. The union bound of proposed receiver will be at form of (37) and (40) with different coefficients. Hence, we can say that the proposed receiver can collect diversity up to order of two, regardless of the underlying constellation.

8 Simulation result

In this section, performance of the proposed receiver will be evaluated. With reference to Fig. 1, downlink of a system with a single decode and forward (DF) relay and two users is considered. Data of users transmit on the same resource (Time) by employing superposition codes at the source. We assume that both users exploit BPSK modulation and power allocation is the solution of (26). We consider a scenario in which SNR in all links is related to \(\bar{\gamma }\), and the mean SNR of each link is adjusted as below in logarithmic scale.

$$\begin{aligned} \gamma _{SR}= & {} \bar{\gamma }\nonumber \\ \gamma _{SD_1}= & {} \gamma _{RD_1}=\bar{\gamma } \nonumber \\ \gamma _{SD_2}= & {} \bar{\gamma }+10 ~~,~~\gamma _{RD_2}=\bar{\gamma } \end{aligned}$$
(41)

In the considered scenario, user 2 has a better direct channel condition, so we can expect that it should allocate less power than the other user. Users decode their symbols based on decision rules (13) and (15). The error probability of both users for (41) presented in Fig. 2, where the solution of (26) is obtained by exhaustive search with the step of 0.01 to find the optimum power allocation. We should consider that the solution of (26) is one of the points where the error probability of users are equal. By considering this fact, we can expect that users should have a tight error probability performance that is perceptible from Fig. 2.

Fig. 2
figure 2

Error probability of proposed receiver with BPSK modulation where signal to noise ratio set as (41) and power allocation is obtained by exhaustive search from (26)

To evaluate the accuracy of the theoretical error probability of users, which presented in (20) and (31), in the second experiment, the error probability of both users is plotted based on theoretical and simulation results. From Fig. 3, we can claim that the calculated error probabilities are accurate. In Fig. 3, it is assumed that the power is adjusted based on the solution of (26), but for all other non-optimum power allocation scenarios, the theoretical and simulation error probabilities will be tight too.

Fig. 3
figure 3

Theoretical and Simulation error probability of users when SNR set as (41)

In Sect. 5, an approximate power allocation for BPSK modulation is proposed as (30). To evaluate the accuracy of proposed power adjustment, in the third experiment, the approximate power allocation applied. First, the power adjusted based on the the solution of (26) by step of 0.001, and then using (30). Error probability of two scenarios presented in Fig. 4.

Fig. 4
figure 4

Error probability of users when SNR set as (41) and power allocation set as (26) and (30)

From this figure, we can conclude that the proposed approximation for optimum power allocation is accurate.

To pinpoint the importance of the power allocation adjustment in NOMA, error probability of users versus \(\alpha \) is presented in Fig. 5 in logarithmic scale when \({\bar{\gamma }}=20\). This figure depicts that the error probability of both users is very sensitive to the power allocation, and must be adjusted precisely to achieve the best performance. Because of the power constraint, for \(\alpha =0\) and \(\alpha =1\) the error probability of one user is very good, but the other user has a high BER. Furthermore, we should notice that allocating equal power to users is not optimal too since in this case the power of main symbol and the interference will be the same which can disrupt the detection procedure.

Fig. 5
figure 5

Error probability of users versus \(\alpha \) when SNR set as (41) and \({\bar{\gamma }}=20\)

To evaluate the influence of relay on the system performance, the scenario of (41) is applied with and without relay presence, and error probability of both scenarios is presented in Fig. 6. The noticeable point which can be concluded from Fig. 6 is that by adding relay to the system, the step of error probability is doubled. It will approve the state that the proposed receiver achieves full diversity.

Fig. 6
figure 6

Error probability with and without relay where SNR set as (41)

To evaluate performance of the system for other modulations, the error probability of proposed receiver is plotted for 4-QAM modulation. Generally, for finding optimum power allocation an exhaustive search should be used. Since we proved that (30) is an accurate approximation for BSPK modulation, we used it for power adjustment. The proposed approximate power has a good performance for 4-QAM and 4-PSK modulation, but its performance will degrade for constellations with higher degree.

Fig. 7
figure 7

Performance of proposed receiver for BPSK and 4-QAM modulations when SNR set as (41)

Fig. 8
figure 8

Performance of the system by changing power at relay

One may argue that the proposed receiver performance is not optimal, due to the constraint of equal powers at BS and the relay. For refuting this argument, the performance of proposed receiver is evaluated for BSPK modulation when power allocation scales at BS and relay are not equal. We should consider that for the feasibility of detection, power allocation in BS and relay should have the same sequence. For instance, if allocated power at BS for user 1 is greater than user 2, the allocated power for user 1 at relay should be greater than user 2 too, otherwise it is not possible to separate symbol of users at destinations. By adding this constraint, the error probability of users can be calculated in the same manner, and the optimal power allocation can be founded by searching among valid regions. BER of both scenarios is plotted in Fig. 8 when SNRs adjusted as (41). From this figure we can claim that the performance of both scenarios is very similar. In Fig. 9 we have compared the BER of proposed detector with the so called Maximum Likelihood (ML) detector which has the best BER performance and is a benchmark for measuring BER performance. Our proposed method’s performance is quite close to ML detector while its complexity is much lower than ML. The ML detector, conducts exhaustive search for detection of symbols through maximization of a nonlinear objective function.

Fig. 9
figure 9

BER Performance of the proposed method and Maximum Likelihood detector for User 1

9 Conclusion

In this paper, we proposed a receiver for downlink of a cooperative system with two users that exploit non-orthogonal multiple access for data transmission. The proposed receiver combines the received signals form direct and cooperative links and employs successive interference cancellation to decode symbol of users. The theoretical error probability of users derived for BPSK modulation. The optimal power allocation problem is defined to minimize the maximum of error probabilities, and an approximate closed form solution was derived. Furthermore, It is proved that the proposed receiver achieves full diversity regardless of underlying constellation.