Keywords

1 Introduction

Recently, attacks made on computer networks have risen dramatically. These attacks are made at various layers in the TCP/IP protocol suite. The attackers act like normal users, generate data and hide their malicious activities under terabytes of data. The monitoring of the network traffic allows to detect malicious activities and perform analysis to differentiate the malicious and non malicious user activities to protect their networks. Detecting malicious activities require intrusion detection systems (IDS). It is critical that an IDS detection mechanism is accurate enough to differentiate between legitimate and malicious traffic that enter and leave the network. The possible results of using an IDS are as follows: detected malicious traffic (real alarms), undetected malicious traffic, legitimate traffic that IDS detect as malicious (false alarms) and legitimate traffic that IDS detect as good.

Intrusion detection is difficult to be accomplished perfectly. With the volume of network traffic rapidly increasing and the number and complexity of network attacks increasing just as quickly, it becomes increasingly difficult for a signature-based intrusion-detection system to keep up with the current threats.

The evaluation performance of intrusion detection systems is a challenging task; it requires a thorough knowledge of techniques relating to different disciplines, especially intrusion detection, methods of attack, networks and systems, technical testing and evaluation.

Lately; more research is done on the evaluation of IDS, we cite some research in this area. In [1,2,3,4] the researchers evaluated the performance of three IDSs in an environment consisted of physical and virtual computers. The experiment results showed that SNORT could have a negative impact on network traffic more than the other two tested IDSs. In [5] a study demonstrated the lack of ability of SNORT IDS to process a number of packets at high speed and the packet drop rate was higher. The researchers introduced a parallel IDS technology to reduce the packet drop rate as a solution. The proposed approach significantly improved SNORT performance [6]. In papers [7, 8] an evaluation of SNORT performance against DDoS. The experiments results show that SNORT packet handling could be improved by using better hardware configurations, but SNORT detection capability was not improved by using better hardware.

In [9], authors have tested and analysed the performance of SNORT and SURICATA. In [10], a comparison analysis of the performance of two open-source intrusion detection systems, SNORT and SURICATA is presented, by evaluating the speed, memory requirements, and accuracy of the detection engines in a variety of experiments. In [11] analysed and implemented the SNORT intrusion detection model in a campus network. In [12] presents a thorough comparison of the performance of SNORT and SURICATA. They examine the performance of both systems as they scale system resources. There are other works that looks at measuring the intrusion detection capability as in tweaking IDS performance as in [13], parallel design of IDS on many-core processors.

In [14], an approach for unifying rule based deep packet inspection and in [15], improving the accuracy of network intrusion detection systems. Whereas in [16], boosting throughput of SNORT NIDS under Linux. As in [17], evaluation studies of three IDS under various attacks and rule sets. In [18], evaluation based in classification of networks attacks [19]. The evaluation based in optimizes and analysis performance of an Intrusion Detection Systems, it is primordial to exploit uniquely the most important and crucial parameters of each features category KDD [20,21,22], etc.

This paper is structured as follows: \(2^{nd}\) Section is devoted to the proposed approach for evaluating performance of the IDSs. Then a presentation and a discussion of the obtained results are shared in Sect. 3. Finally, \(4^{th}\) Section which is dedicated to a conclusion and future works.

2 The Proposed Approach for Evaluating Performances IDS

The experiments consisted of a test bed which compared the performance of both IDSs. Experiment scenarios were designed to make observations and to take measurements. This study demonstrates rigorous, repeatable, quantitative performance comparison of both IDSs. The network traffic for the experiments was produced using network traffic generator. The default rule set of SNORT and SURICATA were used for the experiments.

2.1 Experiment Scenarios

  • Scenario 1 (Consumed resources): The experiment compared the performance of both IDSs by measuring the percentage of CPU (Central Processing Unit), memory utilisation, with different traffic rates.

  • Scenario 2 (Normal traffic accuracy measurements): The experiment was planned to determine the accuracy for both IDS ruleset inspected the network traffic to correctly classify the legitimate traffic network.

  • Scenario 3 (Response to high-speed network traffic): The experiment compared the performance of both IDSs by measuring network packet drop rate, by the transmission of the packets (1460 bytes in size) at different transmission time frames (1, 4, 8 and 16 ms).

2.2 Experiment Network

Fig. 1.
figure 1

Experiment network

Full size image

To perform those tests, the SURICATA version 4.1.1 and the SNORT version 2.9.8.3 are selected. And for experimentations, the network shown in Fig. 1 is created. Four computers were used. Depending on the individual experiment requirements, network packets were produced at varying network speeds with network traffic generator tools. The four computers were connected via a switch CISCO SG350XG-2F10 using 10 Gbps Ethernet links. Each IDS was separately installed on identical computers with default performance parameters and rule set.

2.3 Performance Metrics

The metrics listed below in Table 1 are used to measure the detection accuracy of both IDSs.

Table 1. Description of performance metrics
Full size table

3 Experiment Scenarios Results and Evaluation

3.1 Experiment Scenario One : Consumed Resources

This first experimentation supervises the real-time performance of SNORT and SURICATA while processing at a different normal network speed from a network traffic generator. The rational behind the first experiment is to compare SNORT to SURICATA’s performance. To achieve accurate results, the experiment scenario is tested with packets size of 1460 bytes. These packets were injected to both IDSs with a different network speed. The experiment consisted of a logical network diagram as shown in Fig. 1. Each IDS was separately installed on identical computers with default performance parameters and rule set. A number of tools were used to observe and record the measurements of CPU, memory, network utilisation and the packet drop rate. The following packets were injected as the background traffic ranging from a different network speed (100 Mbps, 250 Mbps, 500 Mbps, 750 Mbps, 1.0 Gpbs, 2.0 Gbps and 4.0 Gbps).

The collected performance data shows that SURICATA’s CPU usage is greater than SNORT’s as explained in Table 2. SURICATA’s CPU utilisation is increased with different traffic rates, while SNORT’s CPU utilisation is comparatively less using the same metrics.

Table 2. (%) CPU and Memory (GB) utilisation for SNORT and SURICATA for different traffic rates
Full size table

The collected performance data shows that SURICATA’s memory usage is greater than SNORT’s as presented in Table 2. SNORT’s memory usage was comparatively less. SURICATA’s memory usage has to do more with the multithreaded architecture.

3.2 Experiment Scenario Tow: Normal Traffic Accuracy Measurements

This experiment is planned to determine how accurately SNORT’s and SURICATA’s rule set in order to inspect the network traffic and correctly classify the non malicious traffic. The metrics listed above in Table 1 are used to measure the detection accuracy of both IDSs.

Table 3. Normal traffic accuracy measurements
Full size table

The second experiment analyses the detection accuracy of SNORT and SURICATA while processing the legitimate network traffic. Both the IDSs were kept at the default setting. The accuracy test was performed using the legitimate network traffic generator which injected UDP, TCP and ICMP packets to both IDSs, the results are shown in Table 3. SURICATA’s false positive rate (FPR) was higher when processing UDP, TCP and ICMP packets than SNORT’s FPR. However, SNORT did not trigger true positive rate (TPR 0%) and false negative rate (FNR 0%) alarms. As compared to SURICATA, this triggered a 41% FNR and 5% TPR. Therefore, SNORT triggered less false positive alarms. While false negative alarms are observed in both IDSs, SNORT’s detection accuracy is found to be superior to SURICATA in this scenario.

3.3 Experiment Scenario Three: IDSs Response to High-Speed Network Traffic

For this third scenario, the packets 1460 bytes in size (\({\cong }100000\) TCP, and \({\cong }100000\) UDP, and \({\cong }100000\) ICMP) are sent at different transmission time frames (1 ms, 4 ms, 8 ms, and 16 ms) for the both systems. Figure 2 shows both IDSs output and obtained results.

Fig. 2.
figure 2

IDSs response to high-speed network traffic

Full size image

As demonstrated in the results shown in Fig. 2, all the sent packets reached their destinations. Both IDSs have analysed almost all packets in incoming traffic when packets are transmitted in 16 ms time frame. But when the speed of transmission is decreased, both IDSs start to drop packets. The collected performance data showed that SNORT’s dropped packets is greater than that of SURICATA for the metrics (1 ms, 4 ms and 8 ms) as in Fig. 2.

In this experiment, it is noticed that for both IDSs; the analysis performance decreased as the speed of transmission is increased. Therfore, the components ability of analysis becomes weaker as the transmission speed is increased.

4 Conclusion

The main contribution of this study is the comparison of the intrusion detection performance of two open source IDSs, namely SNORT and SURICATA. Both are proved to be efficient and high performing IDS, although each one has its own strengths and weaknesses. The analysis of the experiment results shows that SNORT utilises less computational resources to process network traffic whereas SURICATA’s utilisation was higher. Also, SURICATA processes a higher number of packets per second as compared to SNORT, and both IDSs have a high rate of false positives alarms. The obtained results demonstrate a number of signifficant limitations in the use of both IDS.

This work identifies specific and replicable bottlenecks in commonly used implementations IDS in high-speed networks. The obtained results can be taken as a benchmark to improve the performance of these systems in future research work.