1 Introduction

The recent trends of wireless communication have shown attractive features. The mobility support of wireless communication was the core among the other critical solutions. The wireless network relies on the spectrum to communicate for different purposes. However, the spectrum is crowded by increased number of users and diverse applications. Due to the spread of the wireless networks and researches have shown that the spectrum is inefficiently utilized such that they often apply fixed frequency [1].

The unlicensed bands are rarely available and the licensed bands are either underutilized or unoccupied. This problem happens due to the static spectrum assignment and roughly speaking; the dynamic spectrum management simplifies the problem. Hence, the idea of CRN gains promising importance because it permits licensed users to access the spectrum dynamically and the SU can access the licensed and unlicensed spectrum systematically. CRN-based spectrum management is an efficient spectrum utilization, which learns the environment and adjusts its transmission parameters. The key concerns of CRN are, its sensing and cognitivism capabilities. The sensing of a SU can be degraded by fading and hidden terminal problems. This challenge can be solved by cooperative spectrum sensing which involves reporting of sensing results of multiple SUs.

The federal communication commission (FCC) has indicated that only 15 to 85% of the licensed spectrum is utilized [2]. To effectively utilize the radio, an adaptive intelligence has shown fascinating importance to solve the network congestion [3]. With the concept of CRN-based spectrum management, the PUs can transmit at any time with no restrictions, but the SUs has to obtain the benefit of spectrum access when the PUs do not use the corresponding spectrum [4]. A light overhead physical layer network security has been proposed using machine learning [5], but neither the spectrum sensing nor the malicious user detection has been covered. A PU traffic pattern-based opportunistic spectrum handoff has been evaluated to reactively or proactively release the channel depending on the arrival rate of the PU [6], however, the malicious user attack was not considered in the study.

To solve the existence of MUs, heuristic approaches including a genetic algorithm (GA) are applied [7,8,9]. Besides, machine learning (ML) techniques are other good candidates which learns the environment and derives the pattern of the different users. As representative ML-based classification and regression algorithms, there are k-nearest neighbor, decision tree, naïve Bayes, and logistic regression has shown promising performance, [10,11,12].

The performance of cooperative sensing is analyzed with p-norm detector and improved energy detector and the performance is evaluated on a generalized κ-v fading channel, [13]. Throughput enhancement were significant using sequential technique [14], however, the classification of the malicious users was not considered. The machine learning method has been applied to distinguish the PUs and malicious users [15]. However, it was not easy to reliably conclude the type of user as the MU itself can act as legitimate SUs. The machine learning-based sparse coding technique was proposed to detect the PU emulation and jamming attacks, in which the signal obtained from a channel dependent is used to differentiate between a spectrum hole, a legitimate PU, and MU [16]. However, the work fails to distinguish the type of secondary users. The multi-user multi-input multi-output system is evaluated using weighted eigenvalue detection. The research reveals interesting output including improved throughput, reduces energy consumption, and reduces error probability [17]. However, the MUs detection problem has continued a CRN challenge. On the other hand, for defending against a primary user emulation attack, a dogfight approach is used to choose the safest channel [18]. This defense technique uses simple channel sensing and evading the primary user emulation attack. An energy efficient spectrum sensing has also been proposed [19], however, the malicious user detection challenge was not solved in the CRN spectrum management.

Similar work has conducted that incorporates two hidden Markov models [20] to distinguish between malicious and legitimate users. However, the work relies on predefined static pattern. The support vector machine has also shown promising solution to classify the malicious and legitimate user [21, 22] but most of the researches didn’t exploit how the fusion center can operate to effectively manage the malicious users and an unrealistic assumption blurs the classification.

Recently, a network security against the primary user emulsion issue of CRN is modeled [23], however, it lacks to exploit the environmental dynamics of the channel. On the other hand, the type of CRN user classification is studied using support vector machine learning. It uses the signal to noise ratio (SNR) to decide the user classification [24]. However, it didn’t consider the deliberate misleading case of the secondary user because the secondary user can report high or low energy deliberately even when the actual SNR is different.

Though cooperative spectrum sensing provides a promising performance, the existence of malicious users (MUs) deteriorates the performance of the CRN by sending falsified results to a fusion center (FC). Hence, the proposed work focuses on how to combat the MUs to enhance spectrum management. Besides, it is not easy to differentiate the MUs from the legitimate SUs if the MUs’ energy level is very close to the legitimate SUs. For this reason, particle swarm optimization is applied to study the smallest possible distinguishable MUs’ energy report difference from the legitimate SUs.

In general, due to its mobility support, the demand of wireless network is increasing continuously. Besides, when the number of network users increase, the scarcity of the spectrum becomes obvious. However, the spectrum is a limited natural resource and hence a special spectrum management becomes critical. The CRN is one of the intelligent spectrum utilization mechanisms that detects unused spectrum and assigns to the needy without interfering to the legitimate user.

The concept of CRN highly relies on sensing and cognitivism. While the cognitivism focusses on the intelligence in decision making, the sensing is the capability to identify the presence of the PU. It has also indicated that if we try to sense the channel by the central fusion center, there would be high probability that the sensing result could be degraded by fading and hidden terminal problems. For this reason, the significance of cooperative spectrum sensing sounds ideal because this covers spatial distribution. The cooperative spectrum sensing can sense better than a single fusion center. But the significance of cooperative spectrum sensing didn’t come for free. The MUs can deliberately send misleading information about the presence of the PU. Hence, this work proposes support vector machine algorithm to study the MUs’ misleading information. The support vector machine learns the behavior of the MUs and develops a pattern to classify MUs and legitimate users.

Recent research has indicated that simplified assumptions have used to classify the legitimate and malicious users. These includes majority vote, linear searching technique, centralized security (more overhead), and a predefined fixed MU features. On the other hand, researchers assume that the MUs report sensing energy far below or above operating energy report of legitimate SUs, for example [21]. Hence, in this paper, the predefining hypothesis is almost fully avoided and the SVM formulates a pattern by statistically learning to the behavior of the MUs. The optimization tool is also introduced to fully automate the assumptions and classifications.

2 Literature review

It has been several years since the idea of cooperative spectrum sensing becomes attractive. The security challenges, reliable malicious user detection, have made it impractical to apply. Several approaches of malicious user detection have proposed [25, 26], however, either they treat the legitimate or malicious users are well known.

Without complicating the sensing, a compromised sensor node was able to discover using a group voting scheme [27]. On time basis, each node has given a weight and to poll the vote the quality of data transmission and weights are combined. Similarly, a compressed sensing technique is applied [28] in which the MU is removed by processing the signal at the fusion center. A simple majority vote based malicious user detection has applied [29] and the decision is considered anomaly if more than half of the votes suggests malicious user, legitimate user otherwise. This technique provides limited significance because it tends to decide using simple signal strength and it sounds not ideal in tiny sensor networks. Similarly, a simple voting mechanism has proposed for determining the presence of malicious users in the SUs region [30]. In this case, an outlier detection identifies the malicious user using a decentralized approach that uses the geographical correlation of received signal strength in near proximity to the SU. Another simple onion peeling method is introduced as a defensive mechanism against compromising users [31]. This onion peeling method sets-up a threshold to determine the status of the user (MU or LU) and if the current weight exceeds the threshold, the user’s reports are discarded. This cannot classify if the MUs report below threshold weightage.

On a distributed network, a consensus suggestion mechanism is proposed [32]. With many trials, each legitimate SU are made to choose a neighbor with whom to share sensory data. By comparing the received reports to the local mean value, a trustworthy neighbor was identified and the users with different value from the local reference mean are thrown out.

An optimization problem has evaluated with the goal of maximizing CRN energy efficiency while keeping outage probability as a constraint [33]. To attain the CRN energy efficiency, a low-complexity linear searching technique is used, however, it is hard to achieve the goal under multiple MUs scenario. Besides, machine learning technique has applied in CRN in which the frequency band, modulation type, distance, power, and interference influence the system [34, 35]. However, the machine learning power to adapt the real-time radio resources is not explored.

On the other hand, the concept of long short-term memory added with the concept of extreme learning machines [36] is applied for 5G CRN. This avoids the need for large computational cost and it improves the sensing performance because it considers the energy and distance environmental statistics. However, its long time training and centralized security overhead have made it not ideal for 5G CRN networks. The SVM-based spectrum mobility and handover issues have characterized well under dynamic user features [37, 38], however, the impact of malicious users is not studied. Likewise, in a cooperative spectrum sensing, without prior-knowledge on the primary user, the detection performance has improved with the conventional neural network method [39], but the major security drawback of cooperative spectrum sensing is ignored. The support vector machine is also used to play with transmission power and interference to maximize the network throughput [40] but it is hard to classify the different CRN user without predefined fixed features. Finally, the combined method of support vector machine and genetic algorithm (GA) have shown sounding potential to estimate the number of primary users in a CRN [41]. However, the combination of machine learning and optimization didn’t apply to exploit the significance of cooperative sensing and CRN security solutions.

3 Methods

In this paper, a cooperative spectrum sensing method is used to distinguish a malicious user in a cognitive radio network. This cooperative spectrum sensing relies on energy detection to determine the presence or absence of the MUs. In principle, the SU receives the signal of the legitimate user and sends to the fusion center for decision. Malicious users can also send a false information that claims the presence or absence of the legitimate user deliberately to mislead the decision. Hence, the proposed support vector machine learning analyzes the report of the energy and categorizes the energy as SU’s report or MU’s report. The legitimate user and malicious user are categorized using a probability of detection function. The probability of detection and probability of false alarm functions have used to evaluate the user mobility and signal to noise ratio effects. The learning capability of the proposed algorithms statistically studies the behavior of the users based on their energy report.

More specifically, in a wireless network, there are certain number of legitimate SUs and MUs. Both the legitimate SUs and MUs can report their sensing result to the fusion center. The legitimate SUs do this because they are trying to use the channel when the PU are not currently using the channel. The MUs report an energy detection without actually sensing. On applying the classification method, the energy report of all legitimate SUs and MUs is collected. The collected information is used for training. Using the training data, a model is developed. Then once the model is developed, for any energy report from any of the users (legitimate SUs and MUs), we predict the category of the user. Finally, the boundary of the legitimate SUs and MUs are simulated. In the legitimate SUs and MUs classification, the MUs report that the presence or absence of the PUs by sending higher or lower energy levels. In the analysis, the effect of signal to noise ratio on the probability of detection is analyzed. The impact of the MUs on the probability of detection is also studied.

Finally, the malicious users and legitimate users are distinguished by a hyper line. The matlab tool is used to evaluate the learning and to simulate the results for visualization. The numerical evaluations are conducted on 6 MHz bandwidth and 20 microsecond sensing period. More specific experiment parameters and settings are given in Table 1.

Table 1 Simulation parameters

4 System model

In this study, a CRN which characterizes primary users (PUs), secondary users, and a fusion center. The secondary users can be legitimate cognitive radio users (SUs) or malicious users (MUs). Both the malicious and legitimate SUs can perform spectrum sensing and transmission-reception capabilities.

They can also transmit sensing results to the fusion center (FC), as shown in Fig. 1. The SUs, in general, are able to cooperate to determine the presence of the PU in the channel. Each SU can perform local sensing by applying energy detection method of spectrum sensing. In Fig. 1, there are S number of SUs and M number of MUs. There is also a single fusion center, usually a base station, the performs spectrum assignment decisions. The absence and presence of the PU are given by the H0 and H1 terms, respectively.

$$y = \left\{ \begin{gathered} {\text{n; H}}_{{0}} {,} \\ {\text{s}} + {\text{n; H}}_{{1}} , \\ \end{gathered} \right.$$
(1)

where n is the additive white Gaussian noise (AWGN) of the channel, s is the transmitted signal by the PU. To determine the existence of the primary user, the energy detection technique is considered. This can be described by the sum of the energies of the users, given as:

$$E_{i} = \frac{1}{N}\sum\limits_{i = 1}^{N} {\left| {y_{i} } \right|^{2} } ,$$
(2)

where N = 2TW, with T is the sensing duration and W is the bandwidth, and yi denotes the i-th sample of the received signal. Using the Nyquist sampling theorem, the minimum sampling rate (fs) should be atleast twice the operating bandwidth, fs = 2 W. Considering the central limit theorem, if N is large enough (N > 200), the combined signal can be well approximated as a Gaussian random variable under hypotheses H0 and H1, with means μ0 and μ1 and variances σ20 and σ21, which are given by

$$\begin{gathered} {\text{H}}_{0:} \mu_{0 } \, = \,{\text{N,}}\,\mu^{2}_{0} \, = \,{\text{N}}\left( {{\text{y}}\, + \,1} \right) \hfill \\ {\text{H}}_{1:} \mu_{1} \, = N,\,\mu^{2}_{1} \, = \,2{\text{N}}\left( {2{\text{Y}}\, + \,1} \right) \hfill \\ \end{gathered}$$
(3)
Fig. 1
figure 1

Proposed system model

To describe the probability of detection and false alarm, the Gaussian distribution can be used as an approximation to the real-valued random variables with the mean μ and the variance σ2. The mean describes the location of the peak and the variance describes the width of the Gaussian distribution.

$$f(x) = \frac{1}{{\sqrt {2\pi \sigma^{2} } }}e^{{ - \frac{{(x - \mu )^{2} }}{{2\sigma^{2} }}}}$$
(4)

Considering the maximum posteriori, the probability of false alarm (Pf) and probability detection (Pd) can be expressed as (5):

$$\begin{gathered} P_{d} (\lambda ) = P_{r} [y > \lambda |H_{1} ] \hfill \\ P_{f} (\lambda ) = P_{r} [y > \lambda |H_{0} ], \hfill \\ \end{gathered}$$
(5)

where λ is the decision threshold of maximum posteriori detection. If we have a normal variable X with N(µ; σ2), the probability that y > λ is

$$P_{r} [y > \lambda ] = Q\left( {\frac{\lambda - \mu }{\sigma }} \right)$$
(6)

The Q-function Q (x), given in (7), is also used to express the probability of Gaussian random variables. For all real-valued x, Q (x) is defined as the probability of standard normal random variable (zero mean, unit variance) exceeds λ:

$$Q(\lambda ) = \frac{1}{{\sqrt {2\pi } }}\int\limits_{\lambda }^{\infty } {e^{{ - \frac{{t^{2} }}{2}}} } dt = \frac{1}{{\lambda \sqrt {2\pi } }}e^{{ - \frac{{\lambda^{2} }}{2}}}$$
(7)

The Q-function is also directly related to complementary error function (erfc) as

$$Q(\lambda ) = \frac{1}{2}erfc\left( {\frac{\lambda }{\sqrt 2 }} \right)\,{\text{where}}\,erfc(\lambda ) = \frac{2}{\sqrt \pi }\int\limits_{\lambda }^{\infty } {e^{{ - t^{2} }} } dt$$
(8)

Considering the central limit theorem for large number of samples, the Chi-square distribution is approximate as Gaussian distribution and the probabilities of false alarm and detection are, respectively, simplified as [42]:

$$P_{f} = \frac{1}{2}erfc\left[ {\frac{1}{\sqrt 2 }\left( {\frac{{\lambda - N\sigma^{2}_{u} }}{{\sigma^{2}_{u} \sqrt {2N} }}} \right)} \right]$$
(9)
$$P_{d} = \frac{1}{2}erfc\left[ {\frac{1}{\sqrt 2 }\left( {\frac{{\lambda - N(y_{i} + 1)\sigma^{2}_{u} }}{{\sigma^{2}_{u} \sqrt {2N(2y_{i} + 1)} }}} \right)} \right],$$
(10)

where σ2u is the noise variance and yi is the Signal to Noise Ratio of the PU at the ith SU.

5 Proposed machine learning solution

In this study, S number of SUs and M number of MUs are considered, as shown in Fig. 1. It is assumed that the number of legitimate secondary users are more than the number of malicious users, M < S. The SUs listens to the channel for the presence of the PU and they cooperate to sense the activity of the PU channel to the fusion center (FC). The MUs can send false information which damages the significance of the CRN system. The MUs may send deliberate misleading information to the FC. This information can falsely indicate the presence of the PU or the absence of the PU. Hence, the MUs send either higher energy detection to indicate the presence of the PU on the channel, called always yes (AY), or it sends lower energy detection to indicate the absence of the PU, called always no (AN). If sufficient learning data is available, machine learning are fascinating solutions without explicitly programmed.

In this work, machine learning is used to statistically classify the legitimate SUs and MUs. The support vector machine learning-based statistical classification [43] is employed. Statistical machine learning relies on averaging the energy reporting of the users (SUs and MUs) on continuous several observations. The legitimate SUs reports the right sensing results in which the energy of detection can be high or low depending on the received signal level from the PU. However, the MUs reports either higher energy detection (AY) or lower energy detection (AN) irrespective of the PU status. Hence, the proposed statistical machine learning averages the energy reporting and classify the type of the user.

The training data set (D) is described as:

$$D = \left\{ {\left( {E_{i} ,C_{i} } \right)|E_{i} \in \Re ,C_{i} \in \left\{ {1, - 1} \right\}} \right\}$$
(11)

Here the Ci is the class of MUs or SUs indicator vector, and the class “1” and “− 1” represent legitimate SUs and MUs, respectively. The Ei is the energy vector determined by the maximum posteriori energy detection. After the sensing duration is over, the energy reports of all users (MUs and SUs) is applied to the machine learning for building a model, called exploration phase. From the machine learning trained model, a hyper line is formed to classify the MUs and SUs. The FC, then, uses this line to distinguish the class. This line, also called hyperplane, can be formulated by:

$$\begin{gathered} h.x + p = a \hfill \\ h.x + p = - a, \hfill \\ \end{gathered}$$
(12)

where p is a threshold, a is any constant, and h is the weight vector obtained in training phase. In the training, the two different classes are categorized as the MUs and SUs. The pseudocode of the proposed machine learning is summarized as:

  1. 1.

    Initialize the learning

    1. a.

      Initialize the number of SUs and MUs

    2. b.

      Generate sensing reports of SUs and MUs

    3. c.

      Export sensing energy reports

  2. 2.

    Exploration

    1. a.

      Load the training data set

    2. b.

      Develop a model

    3. c.

      Capture model parameters

  3. 3.

    Exploitation

    1. a.

      For any similar data & user

    2. b.

      Predict SUs & MUs boundary

    3. c.

      Scatter the class of each user

    4. d.

      Display visualization

  4. 4.

    End

Many researches assume that the MUs report either quite higher energy than the legitimate SUs or much lower energy than the legitimate SUs. However, the MUs may learn the behavior of the legitimate SUs and they can report an energy detection that closely resembles the SUs. In that case, it is difficult to distinguish the MUs from the SUs. Hence, in this paper, an optimization tool is integrated to learn the smallest possible distinguishable MUs’ energy report deviation from the SUs. To optimize the smallest possible malicious user energy deviation, a PSO is used. This energy deviation is the optimal distinguishable energy that the malicious user can apply to falsely report the presence or absence of the PU. The PSO is characterized as an iterative optimal searching algorithm. PSO is initialized with a group of random particles and searches for the optimal solutions by updating generations. In every iteration, each particle is updated by determining the nearest best values.

The optimization algorithm particles represent the MUs and legitimate SUs. The velocity and position indicates the energy levels of the MUs and legitimate SUs, respectively. The Global best, G(best), and Private best, P(best), represents the current smallest detectable energy level between the MUs and SUs among all of the users. The summary of the PSO algorithm is given in Fig. 2.

Fig. 2
figure 2

PSO working skeleton

6 Results and discussion

To evaluate the performance of the proposed optimization and CRN user classification, MATLAB 2018a is used. The rest of the simulation parameters are given in Table 1.

Before the malicious CRN users detection and FC decision mechanism, the major factors that determine the probability of detection is shown in Fig. 3. The CRN is primarily proposed to alleviate the spectrum scarcity when the number of users are increasing in number and functionality. The ultimate significance of wireless communication is its mobility capability and the CRN assumes mobile users share the licensed spectrum.

Fig. 3
figure 3

SNR effects on the probability of detection

When the users are mobile, they logically receive different signal strength depending on the legitimate SUs location and PU position. Hence, the legitimate SUs can be largely affected by the SNR, as shown in Fig. 3.

On the other hand, mobile users experience different probabilities of detection. The probability of false alarm and the probability of detection is almost directly proportional, as shown in Fig. 4. At the same time, Fig. 4 indicates that different probability of detection can be reported to the FC from different legitimate SUs.

Fig. 4
figure 4

Probability of detection of random SUs

The probability of detection (Pd) is also largely affected by the presence of malicious users. Depending on the attack type (AY or AN), the probability of detection deviates from the right detection. Considering the energy of detection, covered in Figs. 7 and 8, the AY and AN type of attack report higher energy and lower energy, respectively, without sensing the actual presence of the PU. In the presence of the MUs, the average probability of detection violates the right probability of detection, shown in Fig. 5.

Fig. 5
figure 5

The impact of the MUs on the probability of detection

The energy detection reports, Fig. 6, indicates that different CRN users (including the legitimate SUs and MUs) reports different energy levels. Simply collecting the energy level reports at the FC, it is not easy to distinguish the MUs from the legitimate SUs. This diminishes the significance of the CRN system in several years.

Fig. 6
figure 6

Unclassified CRN users (legitimate SUs and MUs)

By applying statistical machine learning, the behaviour of the MUs and legitimate SUs is studied. The AY MUs reports higher energy to deliberately mislead the decision making center, Fig. 7. The proposed machine learning statistically learns which user is energy detection reporting from sensing and which user is reporting without the knowledge of the presence of the PU. The learning averages the energy report of the individual users over 10,000 sample space and it classifies the MUs from the legitimate SUs. Similarly, the AN MUs reports lower energy to deliberately mislead the decision making center, Fig. 8.

Fig. 7
figure 7

CRN users classification (legitimate SUs and AY type of MUs)

Fig. 8
figure 8

CRN users classification (legitimate SUs and AN type of MUs)

Hence, in this work, an optimization tool is used to determine the minimum energy report of the MUs different from the legitimate SUs’ energy report. Several researches assumed that there is a large energy gap between the MUs and legitimate SUs, however, the MUs may learn and apply an energy report quite close to the legitimate SUs. Though the learning itself determines the pattern of the MUs, the optimization exploits the behaviour of the MUs in millions of iterations. The optimization enables the researchers to consider a more realistic scenarios. The machine learning closely studies the MUs and legtimate SUs energy detection pattern and formulates a new model. The new model is derived from exploring the training data set and the model is used to exploit any future predictions.

The proposed machine learning also provides a decision line, Figs. 7 and 8, and the FC easily outlines that the users above the decision line (in Fig. 7) are MUs and the users below the decision line (in Fig. 8) are MUs. This provides an easier decision framework to the fusion center.

The support vector machine has proposed to classify normal SUs and MUs with the range of sensing energies at the legitimate SUs lies on the range of 90–108 [21]. The AN and AY energies are assumed far below and above the legitimate energy ranges. However, the more it is assumed that the MUs do not operate closer to the range of the legitimate user the easier it will be to classify but a little far from the reality. Figures 7 and 9 of [21] indicate that the assumption of the sensing energies of the MUs are around 9. In this paper, we have introduced an optimization algorithm that determines the minimum sensing energy gap between the legitimate and malicious users. Consequently, the proposed solution learns and classifies the legitimate and malicious users even when their sensing energies are close to one as it can be shown in Figs. 7 and 8 of this work.

To sum up, the cooperative spectrum sensing enables ideal solution to sense the existence of the PU. Instead of using another device or employing the unreliable centralized sensing, the cooperative sensing offloads the FC task. Hence, the spectrum management is simplified by applying the cooperative spectrum sensing. However, the cooperative spectrum sensing is highly exposed to malicious user attacks. For this reason, the major contribution of this paper focuses on the MUs detection mechanisms. The machine learning is used to study the characteristics of the MUs and then to determine the pattern of the malicious users. The learning averages the energy detections of the individual users. The legitimate users apply valid spectrum sensing while the MUs send any energy level to the FC without actually sensing the presence or absence of the PU. The proposed machine learning watches the energy report of every user and determines a pattern that characterizes the type of user. The optimization algorithm is also applied to realize the energy reporting by the MUs.

7 Conclusion

To solve the CRN spectrum management challenge, a cooperative spectrum sensing is proposed. The legitimate SUs-based of PU spectrum sensing simplifies the spectrum management of the CRN system. However, this method of spectrum sensing is highly exposed to malicious user attacks. Machine learning is proposed to statistically study the experience of malicious users. Furthermore, the learning classifies the MUs and legitimate SUs with a decision line. Depending on the type of attack (AY or AN), the MUs are classified as above or below the decision line and that region is blacklisted from further accommodation in the spectrum management. The MUs are assumed that they can act quite similar behavior to the legitimate SUs. The simulation results have shown that the proposed method distinguishes the MUs from the legitimate SUs well. The FC can easily decide the spectrum management by considering the decision line to separate the MUs and legitimate SUs.