1 Introduction

Interference in cellular systems is a common issue that affects higher data delivery and reliable QoS. To reduce such performance limiting interference, multicell coordination and cooperation over transmission is employed. Recently Coordinated Multi-Point (CoMP) transmission and reception has been considered as a promising technique to either coordinate or exploit the interference to improve the system throughput and the user fairness. It has been incorporated in the 3GPP Long-Term Evolution Advanced (LTE-A) system to enhance cell edge coverage and spectrum efficiency in order to provide better QoS [1, 2]. The CoMP performs dynamic coordination among multiple geographically separated transmission nodes and its operations are categorized into coordinated beamforming and scheduling (CB/CS) and joint processing (JP) [1]. For CB/CS, the data are transmitted to single user equipment (UE) from only one BS, whereas user scheduling and beamforming decisions are coordinated among BSs belonging to the CoMP cluster set. In JP, the data is transmitted simultaneously to a single UE from two or more BSs in the cooperating set by spatial multiplexing, as illustrated in Fig. 1. Since in JP two or more BSs coordinately transmit signals, the signal quality at the UE is improved significantly at the expense of radio resource [2]. The JP operation is more challenging one, since it exploits the abundant spatial resources provided by the cooperating BSs, where data, channel state information (CSI), scheduling decisions, and precoding vectors need to be shared among BS [3] through central units (CU). In contrast, CB/CS reduces interference by using individual precoding at each BS where only CSI is shared [4]. Since CSI sharing requires a much lower spectrum than sharing data [4], CB/CS needs much lower backhaul (BH) capacity than JP. Coordination for transmission and reception among BSs is characterized by the need of an interconnection among the different nodes in the form of very high speed links known as BH connection. Therefore, the two categories of CoMP operation are differentiated by BH load. These BH links are essential for the success of the cooperative communication, due to the large amount of data that may need to be exchanged among the nodes. In order to achieve the potential cooperation gain for the joint processing schemes, high capacity and low latency BH links are required. However, in realistic system particular attention is needed over BH constraints, which affect the performance of multicell cooperation.

Fig. 1
figure 1

CoMP–joint processing scenario

Extensive studies on transmission mode selection schemes has been proposed in [5, 6] for multicellular systems. However, in reality, the implementation of perfect BS cooperation has the following limitations: BH capacity, BH latency and imperfect CSI. In Refs. [7, 8] CoMP is analyzed under unlimited BH capacity. Based on this assumption, to achieve higher downlink throughput mode selection between CoMP and Non CoMP transmission is proposed [9]. As discussed in [10], cooperative group coding with sharing of partial decoding is more effective with the limited BH capacity. In Ref. [11], for uplink MIMO the estimate–compress-forward approach is investigated over BH constraints. Thus, the BH allocation must be carefully designed for feasible and efficient transmission. The limited bandwidth should be allocated to make full use of the transmission capacity, and it has to be carefully handled to allocate the left over bandwidth in a fair manner. In Ref. [12], the max–min fairness allocation in linear transceiver design problem was proposed, to maximize the sum rate utility for the Multi-User MIMO interfering channel as the number of users becomes large. The power control schemes to provide a higher minimum user data rate as well as greater fairness in a HetNet is proposed in [13]. Power constrained Multicell MISO downlink max–min SINR optimization problem and the virtual uplink min–max SINR optimization problem has been studied in [14]. It should be noted that max–min fair resource allocation provides better user fairness.

As the BH capacity is a major concern in CoMP transmission, attention has to be paid on limited BH utilization among cooperative BSs. Therefore, selecting the significant number of cooperative BSs from the cooperative cluster is incorporated that will reduce the BH burden and further enhance the performance of CoMP transmission under practical constraints such as power reduction. In this paper, a dynamic cooperative base station selection scheme has been proposed to reduce the BH load and at the same time guarantee the QoS for all users under the max–min fairness criterion in downlink cellular network. In the proposed scheme, each user in the cooperative cell is served with the essential number of BSs selected dynamically for cooperation, which involves the effective utilization of available BH capacity while consuming less power due to minimal coordinated BS transmission. However, the proposed scheme is similar to conventional CoMP operation, unlike with all cooperative BS, only the essential number of selected BS performs cooperative transmission to achieve the required QoS for cell edge users. The sum rate of the CoMP system is maximized with max–min fairness allocation under a given BH capacity constraint. In this paper, CoMP–JP is considered, for convenience it is referred as CoMP. The rest of this paper is organized as follows. In Sect. 2, the system model with CoMP and Non CoMP is presented. In Sect. 3 proposed scheme for BS selection is explained. Simulation results and discussion are provided in Sect. 4 followed by conclusion in Sect. 5.

2 System Model

Consider the cooperative downlink of a multicellular multiuser MIMO communication system composed with M BSs and K co channel cell edge UE. Each BS is equipped with N b transmit antennas and each UE employs N r receive antennas. As shown in Fig. 2, each BS is connected with CU via a digital bi-directional backhaul link with limited capacity C BH in each direction and collaboratively serves the K UEs thus characterizing a CoMP structure [15]. Assume that the BH links are wired and lossless; hence the BH and the radio access network occupy orthogonal resources. Each BS is assumed to share the data and the mean channel quality indicators (CQIs) for the targeted active UEs via a reliable BH links. The multi-user scheduling and precoding is performed by CU via high speed BH connections which connects to each BS and synchronization among the BSs are assumed. The downlink information, control data and precoder are communicated to each BS through the X2 interface, where all BSs serve the UEs cooperatively. The downlink transmission to each user can be CoMP or Non CoMP and this mode of transmission of each UE is mainly depending on its locations, channel conditions, signaling overhead, and BH capacity. In this paper, perfect CSI at each BS is assumed.

Fig. 2
figure 2

System Model

\(H_{k}^{m}\) is the CoMP channel matrix from BS m to user k, which is the result of contribution of frequency selective Rayleigh fading, shadowing, and path loss. Small scale fading channel vector is denoted by \(C_{k}^{m} \in {\mathbb{C}}^{Nr \times Nb}\), each entry of which is complex Gaussian random variables where (p, q) entry of \(C_{k}^{m}\) denotes the path gain from the qth antenna of the mth BS to the pth antenna of the kth user and is assumed as independent and identically distributed. The element of channel matrix is given by \(H_{k}^{m} (p,q) = \alpha_{pk,qm} C_{k}^{m} (p,q)\) where \(\alpha_{pk,qm}\) is the large-scale fading channel gain from BS m to user k.

2.1 CoMP Mode Transmission

In a cooperative system, received signal \(Y_{k}^{C} \in {\mathbb{C}}^{Nr \times 1}\) from the cooperative BS m to user k can be expressed [9] as

$$Y_{k}^{C} = \sum\limits_{m = 1}^{M} {H_{k}^{m} V_{k}^{m} \sqrt {p_{k}^{m} } X_{k} } + \sum\limits_{j = 1,j \ne k,m = 1}^{MK} {H_{k}^{m} V_{{c_{j} }}^{m} \sqrt {p_{{c_{j} }}^{m} } X_{{c_{j} }} } + n_{k}$$
(1)

where \(V_{k}^{m} ,V_{{c_{j} }}^{m} \in {\mathbb{C}}^{{MN_{b} \times 1}}\) are the global precoding vectors for the user k and user c j . Zero-forcing (ZF) precoder is considered due to its low complexity and the matrix is given by \(V_{k}^{m} = H_{k}^{m} \left( {H_{k}^{{m^{H} }} H_{k}^{m} } \right)^{ - 1} P_{k}\), where \(P_{k} = diag\left\{ {p_{k,1} , \ldots ,p_{{k,N_{s} }} } \right\}\), represents the downlink power allocation matrix of the UE k and \(p_{k,i}\) refers to the downlink power allocated to the ith stream of the kth user.

\(X_{k} ,X_{{c_{j} }} \in {\mathbb{C}}^{{N_{s} \times 1}}\) are the data intended for user k and user c j respectively, that is assumed to be i.i.d. and \(E\left\{ {X_{k} X_{k}^{H} } \right\} = E\left\{ {X_{{c_{j} }} X_{{c_{j} }}^{H} } \right\} = I_{{N_{s} }}\), Ns denotes the number of data streams for the kth user; where \(E\{ .\}\) represents expectation.

The second term \(IUI_{{c_{j} }}^{{}} \triangleq H_{k}^{m} V_{{c_{j} }}^{m} X_{{c_{j} }}\) represents the inter-user interference (IUI) from user c j to user k.

\(n_{k} \in {\mathbb{C}}^{{N_{r} \times 1}}\) denotes the noise at the user k, which is a zero mean complex additive white Gaussian random variable with covariance matrix \(\sigma_{{}}^{2} I_{Nr}\).

The received signal-to-noise-plus-interference ratio (SINR) of user is given by

$$\gamma_{k}^{C} = \frac{{P_{k}^{C} }}{{\sigma_{{}}^{2} }}$$
(2)

The net downlink throughput of the user k under CoMP transmission can be written as

$$\Re_{k}^{C} = C_{BH} \log \left( {1 + \gamma_{{_{k} }}^{C} } \right)$$
(3)

where \(C_{BH}\) [Hz] is the backhaul capacity.

2.2 Non CoMP Mode Transmission

Traditionally, in cellular system UEs are served by single BS known as Non CoMP transmission mode, where each BS serves its K active users located in its own serving cell. In this mode, each user receives the intended signal from its local BS, the signals received from the rest of the BSs are in the form of inter-cell interference that degrades the system performance. However, this mode of transmission is selected for user at cell center region due to less signaling overhead and BH load.

The element of channel matrix for Non CoMP is given by \(G_{k}^{m} (p,q) = \alpha_{pk,qm}^{m} C_{k}^{m} (p,q)\). The received signal \(Y_{k}^{NC}\) of user k in the cell m can be expressed as

$$Y_{k}^{NC} = \sum\limits_{i = 1}^{M} {G_{k}^{i} V_{k}^{i} \sqrt {P_{k}^{i} } X_{i}^{{}} } + n_{k}$$
(4)
$$Y_{k}^{NC} = G_{k}^{m} V_{k}^{m} \sqrt {p_{k}^{m} } X_{k} + \sum\limits_{j = 1,j \ne k}^{K - 1} {G_{k}^{m} V_{{s_{j} }}^{m} \sqrt {p_{{s_{j} }}^{m} } X_{{s_{j} }} } + \sum\limits_{i = 1,i \ne m}^{M} {G_{k}^{i} V_{k}^{i} \sqrt {p_{k}^{i} } X_{i} } + n_{k}$$
(5)

where \(V_{k}^{i} \in {\mathbb{C}}^{{N_{b} \times M}}\) is the precoding matrix at BS \(i\).

\(X_{i} \in {\mathbb{C}}^{M \times 1}\) is the intended data vector at BS i for its K serving users.

\(G_{k}^{m} V_{sj}^{m} X_{sj}\) represents the IUI from user s j to user k; and \(G_{k}^{i} V_{k}^{i} X_{i}\) where i ≠ m represents the inter cell interference from BS i to user k.

The zero forcing precoding matrix is \(V_{k}^{m} = G_{k}^{m} \left( {G_{k}^{{m^{H} }} G_{k}^{m} } \right)^{ - 1} P_{k}\), where \(G_{k}^{m}\) is the Non CoMP channel matrix of the K active users in the cell m and \(P_{k}^{i} = diag\{ p_{k,1} , \ldots ,p_{k,MK} \}\) is the power allocation matrix.

\(n_{k} \in {\mathbb{C}}^{Nr \times 1}\) denotes the noise at the user k, which is a zero mean complex additive white Gaussian random variable with covariance matrix \(\sigma_{{}}^{2} I_{Nr}\).

The received SINR of user under Non CoMP is written as

$$\gamma_{k}^{NC} = \frac{{\left( {\alpha_{pk,qm}^{m} } \right)^{2} P_{k}^{NC} }}{{P_{K} \sum\limits_{i \ne m}^{M} {\left( {\alpha_{k}^{i} } \right)^{2} } \sigma_{e}^{2} + \sigma^{2} }}$$
(6)

The net downlink throughput of the user k under Non CoMP transmission is given as

$$\Re_{k}^{NC} = C_{BH} \log \left( {1 + \gamma_{{_{k} }}^{NC} } \right)$$
(7)

3 Proposed Dynamic Cooperative Base Station Selection (DCBS) Scheme

In this section, the dynamic selection of BSs for cooperative transmission has been discussed. The proposed algorithm is dynamic since the mode selection is achieved at each transmit time interval (TTI) denoted as \(\Im\).

3.1 Proposed Cooperative Transmission Scheme

A closed form cooperative BS selection rule is developed which depends on the location of user and the number of cooperative BSs, transmit antennas, BH capacity and cell edge SINR. The proposed scheme, initially determine the user as either cell center user or cell edge user by estimating the net throughput of the user with its serving BS and cooperating BS in the cluster. The BS is selected by the CU, based on the feedback of CSI and PMI. At each TTI, each user is allowed to select CoMP transmission mode when \(\Re_{k}^{C} > \Re_{k}^{NC}\) otherwise the user is served through single BS as a cell center user when \(\Re_{k}^{C} < \Re_{k}^{NC}\). To maximize the overall sum rate of the system with limited BH capacity, each CoMP user is allowed to select the significant number of cooperative BSs from cooperative cluster when \(\Re_{k}^{C} > \Re_{k}^{thres}\),where \(N \le M\) and the remaining cooperative cells are muted. After every TTI the user status is checked through CQI, the BSs are selected based on the requirement to achieve required QoS of CoMP with effective utilization of BH capacity and at the same time, the mode of transmission has to be estimated dynamically.

Let \(r^{M}\) denotes the transmit rate of the each coordinated BS whose indices are M and the total transmit power at each BS is denoted as P t . In CoMP mode, the sum power of all cooperative BS is MP t which is equally allocated to the MK active users. From Eq. (2), the SINR of CoMP transmission can be written using [9] as

$$\gamma_{k}^{C} \approx \frac{{P_{t} \left| {H_{k}^{m} } \right|\delta_{MK - 1} }}{{K\sigma^{2} }}$$
$$= \frac{{P_{t} N_{b} \left( {\sum\nolimits_{i = 1}^{M} {\alpha_{u}^{{i^{2} }} } } \right)\delta_{MK - 1} }}{{K\sigma^{2} }}$$
(8)

where \(\delta_{MK - 1} \mathop = \limits^{\Delta } \sin^{2} \left( {\theta_{k}^{C} } \right)\) is the angle between the channel of user \(k\) and a subspace spanned by the co-scheduled user channels \(MK - 1\), whose value is between 0 and 1. Similarly, for Non CoMP transmission Eq. (6) the SINR is given by

$$\gamma_{k}^{NC} \approx \frac{{P_{t} N_{b} \left| {G_{k}^{m} } \right|\{ \lambda_{K - 1} \} }}{{K\left( {P_{t} \sum\limits_{i \ne m}^{M} {\left( {\alpha_{k}^{i} } \right)^{2} } \sigma_{e}^{2} + \sigma^{2} } \right)}}$$
(9)

where \(\lambda_{K - 1} \mathop = \limits^{\Delta } \sin^{2} \left( {\theta_{k}^{NC} } \right)\) is the orthogonality between the channel of user k and a subspace spanned by the co-scheduled user channels \(K - 1\), whose value is between 0 and 1.For proposed selection scheme, for selected BSs, the data from other BSs are muted, i.e.\(X^{M} = 0,M \ge N\quad forM \ne N\). Therefore, Eq. (1) the received signal \(Y_{k}^{DCBS} \in {\mathbb{C}}^{Nr \times 1}\) from the selected N cooperative BSs to user k can be rewritten as

$$Y_{k}^{DCBS} = \sum\limits_{n = 1}^{N} {H_{k}^{n} V_{k}^{n} \sqrt {p_{k}^{n} } X_{k} } + \sum\limits_{j = 1,j \ne k,n = 1}^{NK} {H_{k}^{n} V_{{c_{j} }}^{n} \sqrt {p_{{c_{j} }}^{n} } X_{{c_{j} }} } + n_{k}$$
(10)

From Eq. (8), the SINR for proposed DCBS scheme with selected N BSs from M cooperative BS cluster is expressed as

$$\gamma_{k}^{DCBS} = \frac{{P_{t} N_{b} \left( {\sum\nolimits_{i = 1}^{N} {\alpha_{k}^{{i^{2} }} } } \right)\delta_{NK - 1} }}{{K\sigma^{2} }}$$
(11)

The net downlink throughput in the proposed DCBS transmission is given as

$$\Re_{k}^{DCBS} = C_{BH} \log \left( {1 + \gamma_{k}^{DCBS} } \right)$$
(12)

3.2 Max–Min Criterion for Proposed Cooperative Transmission Scheme

In this paper, as BH capacity is limited, problem has to be formulated to ensure the effective use of BH load to achieve the system sum rate as much as possible. Moreover, the sum rate maximization can achieve high network throughput, the max–min fair strategy is used which maximizes the minimum SINR of all users (equivalent to the minimum rate) in the network. The problem is formulated for proposed scheme [12] as

$$\begin{aligned} & \mathop {\hbox{max} }\limits_{\begin{subarray}{l} \{ v_{k}^{m} ,r_{k}^{m} \} \\ 1 \le N \le M \end{subarray} } U\left( {\gamma_{1} ,\gamma_{2} , \ldots ,\gamma_{k} , \ldots ,\gamma_{K} } \right) \\ & s.t.\quad \left| {v_{k}^{m} } \right|^{2} = 1,\left| {r_{k}^{m} } \right| \le C_{BH} ,\quad k \in K \\ & \quad \;\;\;\Sigma _{i = 1}^{K} P_{i} = P_{t} \\ \end{aligned}$$
(13)

The sum rate maximization of proposed scheme is expressed as

$$U\left( {\gamma_{1} ,\gamma_{2} , \ldots ,\gamma_{k} , \ldots \gamma_{K} } \right) = \sum\limits_{k \in K} {C_{BH}^{{}} } \log \left( {1 + \gamma_{k}^{DCBS} } \right)$$
(14)

The max–min rate fairness resource allocation aims at maximizing the minimum SINR of all the users and can be expressed as follows

$$\begin{aligned} & \mathop {\hbox{max} }\limits_{\begin{subarray}{l} \{ v_{k}^{m} ,r_{k}^{m} \} \\ 1 \le N \le M \end{subarray} } {\rm Z}\left( {v_{k}^{m} ,r_{k}^{m} } \right) = \mathop {\hbox{min} }\limits_{k \in K} \left\{ {\gamma_{k}^{DCBS} } \right\} \\ & s.t.\quad \left| {v_{k}^{m} } \right|^{2} = 1,\left| {r_{k}^{m} } \right| \le C_{BH} ,\quad k \in K \\ & \quad \;\;\;\Sigma _{i = 1}^{K} P_{i} = P_{t} \\ \end{aligned}$$
(15)

The proposed dynamic cooperative base station selection algorithm is summarized in Table 1.

Table 1 Dynamic cooperative base station selection algorithm

4 Simulation Results and Discussion

In this section, the performance of proposed scheme under different backhaul capacity is investigated and compared with other transmission modes in terms of both sum rate and max–min rate for cell edge user. Each sum rate is the average over 1000 random channel realizations. A cooperative cluster of three hexagonal cells is considered with inter-BS distances of 2 km. The four users are randomly located and associated with nearest BSs. Fifteen BSs are considered in the cluster and ten BSs are selected for CoMP transmission based on DCBS scheme. The simulation parameters [16] are summarized in Table 2.

Table 2 Simulation parameters

Figure 3 depicts the sum rate (bits per channel use) performance against BH capacity with average SINR fixed at 10 dB. The proposed DCBS scheme outperforms CoMP and Non CoMP mode by a significant value with efficient backhaul utilization. This is because in Non CoMP mode, only single-serving BS is used and interference from other BSs are treated as noise which degrades the performance, also no cooperative diversity is exploited. In CoMP mode, the efficient BS cooperation is achieved by exchanging the control information via the BH link at each stage, which reduces the interference and enhances users’ transmission rates. However, as many number of cooperative BS are involved in CoMP transmission, larger BH capacity is required. On the other hand, in proposed DCBS the users are served with selective number of cooperative BSs hence, outperforms other modes by a large value. This is due to not only taking advantage of interference cancellation, but also efficiently exploiting the selected BS cooperation by exchange of reduced control information through available BH load than CoMP mode. Moreover, DCBS performance is improved through the BS-selection gain and therefore, the users’ transmission rate is increased. It is seen that as the BH capacity grows, the sum rate increases due to more users’ share control information for interference cancellation at each stage. The growth is rapid at the beginning, then slows down and finally, when the BH capacity is high, the sum rate saturates.

Fig. 3
figure 3

Impact of backhaul capacity C BH on Sum rate

Figure 4 illustrates the sum rate performance against SINR with various BH capacity. It is found that the DCBS scheme outperforms both CoMP and Non CoMP mode. The cooperative transmission in CoMP mode requires an additional phase of interactions among coordinating BSs to collectively design the transmitting parameters and/or exchange users’ data, which requires large BH rates and the delivered sum rate to end users largely depends on backhaul rates. In Non CoMP due to no cooperation and signal diversity, intercell interference prevents the cell edge users from achieving their required sum rates. For example, at C BH  = 4 b/s/Hz to provide SINR about 10 dB the proposed DCBS requires a sum rate of 12 (bpcu) which is 2 (bpcu) higher than CoMP mode and 6 (bpcu) higher than Non CoMP mode.

Fig. 4
figure 4

Impact of SINR on Sum rate at various C BH

Figure 5 shows the total transmitted power of various modes against SINR per user. It is clear that the proposed DCBS outperforms CoMP in terms of power consumption. Since, in CoMP mode both data and control information is being shared simultaneously within all BSs in the cluster, which requires additional transmit power for circulating information among all coordinating BSs. In contrast, the proposed base station selection scheme proceeds only with selected number of BSs,the effective exploitation of inter cell interference and benefits from the signal diversity, the cell edge users can attain their required SINR with significantly low power without sacrificing the QoS. It is observed that to achieve SINR of 10 dB, the proposed scheme imposes less power than conventional CoMP mode. In case of Non CoMP transmission as single BS is involved, the circulation of information is reduced. Hence, lesser power is utilized than other modes. However, the cause of increase in total power is to tackle the interference from neighbour cells and path loss.

Fig. 5
figure 5

Total transmitted power of each transmission mode against SINR

The minimum rate for different transmission scheme at C BH  = 3 b/s/Hz is given in Fig. 6. Compared with other modes the CoMP transmission offers larger minimum rate due to more BS is involved in cooperation. Moreover, cooperation in proposed scheme is achieved through the selected number of BSs, hence interference from other cell limits its minimum rate. Finally, the Non CoMP mode has a smaller min rate than other modes since the users served with only one BS, the cell edge users suffer from severe interference from neighbor cells.

Fig. 6
figure 6

Minimum rate for different transmission modes at fixed C BH  = 3 b/s/Hz

Figure 7 illustrates the impact of C BH on minimum rate allocation for the proposed DCBS scheme. Larger BH capacity will support large data for cooperation. When the BH capacity is larger, the selected BSs of each user can share more data cooperatively for interference cancellation, which offers large rate recommendations. Therefore, as the BH capacity increases the allocation of minimum rate increases based on maximizing the minimum transmission rate and guarantees that the every node to be allocated some reasonable amount of bandwidth.

Fig. 7
figure 7

Comparison of Minimum rate versus SINR with different C BH of the proposed DCBS scheme

5 Conclusion

In this paper, a dynamic cooperative BS selection scheme has been proposed to achieve the effective utilization of the limited backhaul capacity and lesser power in downlink cooperative transmission as well as max–min fair rate allocation is incorporated to provide better fairness. The proposed algorithm dynamically selects the essential number of cooperative BS in order to reduce the BH burden while maintaining required QoS for active users in coordinating cells. The selection rule is in closed form, which has an explicit relationship with the average channel gains of each user and various system parameters. Simulation results showed that the proposed scheme maximizes the sum rate of the cell edge users with limited BH load based on max–min fairness allocation and makes a good tradeoff between BH capacity and system performance with better QoS when compared with other transmission schemes such as CoMP and Non CoMP.