Abstract
In this chapter, we survey existing knowledge on the evaluation of security mechanisms by defining an evaluation design space that puts existing work into a common context. Given the significant amount of existing practical and theoretical work, the presented systematization is beneficial for improving the general understanding of the topic by providing an overview of the current state of the field. The evaluation design space that we present is structured into three parts, that is, workload, metrics, and measurement methodology—the standard components of any system evaluation scenario. The discussions in this chapter are relevant for the evaluation of a wide spectrum of security mechanisms, such as firewalls and AC systems.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
References
Allalouf, M., Ben-Yehuda, M., Satran, J., & Segall, I. (2010). Block storage listener for detecting file-level intrusions. In Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST 2010), Incline Village, NV (pp. 1–12). Washington, DC: IEEE Computer Society.
Alserhani, F., Akhlaq, M., Awan, I. U., Cullen, A. J., & Mirchandani, P. (2010). MARS: Multi-stage attack recognition system. In Proceedings of the 24th IEEE International Conference on Advanced Information Networking and Applications (AINA 2010), Perth, WA (pp. 753–759). Washington, DC: IEEE Computer Society.
Arlat, J., Costes, A., Crouzet, Y., Laprie, J.-C., & Powell, D. (1993). Fault injection and dependability evaluation of fault-tolerant systems. IEEE Transactions on Computers, 42(8), 913–923.
Avritzer, A., Tanikella, R., James, K., Cole, R. G., & Weyuker, E. J. (2010). Monitoring for security intrusion using performance signatures. In Proceedings of the First Joint WOSP/SIPEW International Conference on Performance Engineering (ICPE 2010), San Jose, CA (pp. 93–104). New York, NY: ACM.
Axelsson, S. (2000). The base-rate fallacy and the difficulty of intrusion detection. ACM Transactions on Information and System Security, 3(3), 186–205.
Bharadwaja, S., Sun, W., Niamat, M., & Shen, F. (2011). Collabra: A Xen hypervisor based collaborative intrusion detection system. In Proceedings of the 2011 Eighth International Conference on Information Technology: New Generations (ITNG 2011), Las Vegas, NV (pp. 695–700). Washington, DC: IEEE Computer Society.
Carreira, J., Madeira, H., & Silva, J. G. (1998). Xception: A technique for the experimental evaluation of dependability in modern computers. IEEE Transactions on Software Engineering, 24(2), 125–136.
Debar, H., Dacier, M., Wespi, A., & Lampart, S. I. (1998). An Experimentation Workbench for Intrusion Detection systems. Technical report. RZ 2998. IBM T.J. Watson Research Center.
Doddapaneni, K., Ever, E., Gemikonakli, O., Mostarda, L., & Navarra, A. (2012). Effects of IDSs on the WSNs lifetime: Evidence of the need of new approaches. In Proceedings of the 2012 IEEE 11th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom 2012), Liverpool (pp. 907–912). Piscataway, New Jersey: IEEE.
Dreger, H., Feldmann, A., Paxson, V., & Sommer, R. (2008). Predicting the resource consumption of network intrusion detection systems. In Proceedings of the 2008 ACM International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS 2008), Annapolis, MD. ACM SIGMETRICS Performance Evaluation Review (Vol. 36, pp. 437–438). New York, NY: ACM.
Dumitras, T., Shou, D. (2011). Toward a standard benchmark for computer security research: The world-wide intelligence network environment (WINE). In Proceedings of the First Workshop on Building Analysis Datasets and Gathering Experience Returns for Security (BADGERS 2011), Salzburg (pp. 89–96). New York, NY: ACM.
Dunlap, G. W., King, S. T., Cinar, S., Basrai, M. A., & Chen, P. M. (2002). ReVirt: Enabling intrusion analysis through virtual-machine logging and replay. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI), Boston. ACM SIGOPS Operating Systems Review (Vol. 36, pp. 211–224). New York, NY: ACM.
Durães, J. A., & Madeira, H. (2003). Definition of software fault emulation operators: A field data study. In Proceedings of the 2003 International Conference on Dependable Systems and Networks (DSN 2003), San Francisco, CA (pp. 105–114). Washington, DC: IEEE Computer Society.
Durães, J. A., & Madeira, H. (2006). Emulation of software faults: A field data study and a practical approach. IEEE Transactions on Software Engineering, 32(11), 849–867.
Durst, R., Champion, T., Witten, B., Miller E., & Spagnuolo, L. (1999). Testing and evaluating computer intrusion detection systems. Communications of the ACM, 42(7), 53–61.
Engen, V., Vincent, J., & Phalp, K. (2011). Exploring discrepancies in findings obtained with the KDD Cup’99 data set. Intelligent Data Analysis, 15(2), 251–276.
Fonseca, J., & Vieira, M. (2008). Mapping software faults with web security vulnerabilities. In Proceedings of the 2008 IEEE International Conference on Dependable Systems and Networks (DSN 2008), Anchorage, AK (pp. 257–266). Washington, DC: IEEE Computer Society.
Fonseca, J., Vieira, M., & Madeira, H. (2009). Vulnerability and attack injection for web applications. In Proceedings of the 2009 IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2009), Lisbon (pp. 93–102). Washington, DC: IEEE Computer Society.
Gad El Rab, M. (2008). Evaluation des Systèmes de détection d’intrusion. PhD Thesis. Toulouse: Université Paul Sabatier - Toulouse III.
Gaffney, J., & Ulvila, J. (2001). Evaluation of intrusion detectors: A decision theory approach. In Proceedings of the 2001 IEEE Symposium on Security and Privacy, Oakland, CA (pp. 50–61). IEEE: Piscataway, New Jersey.
Görnitz, N., Kloft, M., Rieck, K., & Brefeld, U. (2009). Active learning for network intrusion detection. In Proceedings of the 2nd ACM Workshop on Security and Artificial Intelligence (AISec 2009), Chicago, Illinois (pp. 47–54). New York, NY: ACM.
Griffin, J. L., Pennington, A., Bucy, J. S., Choundappan, D., Muralidharan, N., & Ganger, G. R. (2003). On the Feasibility of Intrusion Detection Inside Workstation Disks. Technical Report CMU-PDL-03-106. Pittsburgh: Parallel Data Laboratory, Carnegie Mellon University.
Gu, G., Fogla, P., Dagon, D., Lee, W., & Skorić, B. (2006). Measuring intrusion detection capability: An information-theoretic approach. In Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security (ASIACCS 2006), Taipei (pp. 90–101). New York, NY: ACM.
Hall, M., & Wiley, K. (2002). Capacity verification for high speed network intrusion detection systems. In A. Wespi, G. Vigna & L. Deri (Eds.), RAID 2002—Proceedings of 5th International Symposium Recent Advances in Intrusion Detection, Zurich. Lecture Notes in Computer Science (Vol. 2516, pp. 239–251). Berlin: Springer.
Hassanzadeh, A., & Stoleru, R. (2011). Towards optimal monitoring in cooperative IDS for resource constrained wireless networks. In Proceedings of 20th International Conference on Computer Communications and Networks (ICCCN 2011) (pp. 1–8), Maui. Piscataway, New Jersey: IEEE.
IBM (2012). IBM X-Force 2012 Mid-Year Trend and Risk Report.
Jin, H., Xiang, G., Zhao, F., Zou, D., Li, M., & Shi, L. (2009). VMFence: A customized intrusion prevention system in distributed virtual computing environment. In Proceedings of the 3rd International Conference on Ubiquitous Information Management and Communication (ICUIMC 2009), Suwon (pp. 391–399). New York, NY: ACM.
Jin, H., Xiang, G., Zou, D., Wu, S., Zhao, F., Li, M., et al. (2013). A VMM-based intrusion prevention system in cloud computing environment. The Journal of Supercomputing, 66(3), 1133–1151.
Katcher, J. (1997). PostMark: A New File System Benchmark. Technical Report TR3022. Sunnyvale: Network Appliance.
Komlodi, A., Goodall, J. R., & Lutters, W. G. (2004). An information visualization framework for intrusion detection. In CHI’04 extended abstracts on human factors in computing systems, Vienna (p. 1743). New York, NY: ACM.
Kruegel, C., Valeur, F., & Vigna, G. (2005). Intrusion detection and correlation—Challenges and solutions. Advances in Information Security (Vol. 14). New York: Springer US.
Leita, C., Dacier, M., & Massicotte, F. (2006). Automatic handling of protocol dependencies and reaction to 0-day attacks with ScriptGen based honeypots. In D. Zamboni & C. Kruegel (Eds.), RAID 2006, Proceedings of 9th International Symposium Recent Advances in Intrusion Detection. Lecture Notes in Computer Science (Vol. 4219, pp. 185–205). Berlin: Springer.
Lombardi, F., & Di Pietro, R. (2011). Secure virtualization for cloud computing. Journal of Network and Computer Applications, 34(4), 1113–1122.
Maynor, D., Mookhey, K. K., Cervini, J., Roslan, F., & Beaver, K. (2007). Metasploit toolkit for penetration testing exploit development, and vulnerability research. Rockland, MA: Syngress Publishing.
McHugh, J. (2000). Testing intrusion detection systems: A critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory. ACM Transactions on Information and System Security, 3(4), 262–294.
Mell, P., Hu, V., Lippmann, R., Haines, J., & Zissman, M. (2003). An Overview of Issues in Testing Intrusion Detection Systems. NIST Interagency/Internal Report (NISTIR) 7007. Gaithersburg: National Institute of Standards and Technology (NIST).
Meng, Y., & Li, W. (2012). Adaptive character frequency-based exclusive signature matching scheme in distributed intrusion detection environment. In Proceedings of the IEEE 11th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom 2012), Liverpool (pp. 223–230). Piscataway, New Jersey: IEEE.
Milenkoski, A. (2016). Evaluation of Intrusion Detection Systems in Virtualized Environments. PhD Thesis. Würzburg: University of Würzburg.
Milenkoski, A., Payne, B. D., Antunes, N., Vieira, M., Kounev, S., Avritzer, A., et al. (2015). Evaluation of intrusion detection systems in virtualized environments using attack injection. In H. Bos, F. Monrose & G. Blanc (Eds.), RAID 2015—Proceedings of the 18th International Symposium Research in Attacks, Intrusions, and Defenses, Kyoto. Lecture Notes in Computer Science (Vol. 9404). Berlin: Springer.
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A., & Payne, B. D. (2015). Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys, 48(1), 12:1–12:41.
MIT Lincoln Laboratory (1999). 1999 DARPA intrusion detection evaluation dataset. https://www.ll.mit.edu/r-d/datasets/1999-darpa-intrusion-detection-evaluation-dataset
Mohammed, N., Otrok, H., Wang, L., Debbabi, M., & Bhattacharya, P. (2011). Mechanism design-based secure leader election model for intrusion detection in MANET. IEEE Transactions on Dependable and Secure Computing, 8(1), 89–103.
NSS Labs (2010). Network Intrusion Prevention System Test Methodology v.6.1. http://www.nsslabs.com/assets/Methodologies/nss2010
Patil, S., Kashyap, A., Sivathanu, G., & Zadok, E. (2004). FS: An In-Kernel integrity checker and intrusion detection file system. In Proceedings of the 18th USENIX Conference on System Administration (LISA 2004), Atlanta (pp. 67–78). Berkeley, CA: USENIX Association.
Raja, N., Arulanandam, K., & Rajeswari, B. (2012). Two-level packet inspection using sequential differentiate method. In Proceedings of the International Conference on Advances in Computing and Communications (ICACC 2012), Cochin (pp. 42–45). Piscataway, New Jersey: IEEE.
Reeves, J., Ramaswamy, A., Locasto, M., Bratus, S., & Smith, S. (2012). Intrusion detection for resource-constrained embedded control systems in the power grid. International Journal of Critical Infrastructure Protection, 5(2), 74–83.
Riley, R., Jiang, X., & Xu, D. (2008). Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In R. Lippmann, E. Kirda & A. Trachtenberg (Eds.), RAID 2008— Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection, Cambridge. Lecture Notes in Computer Science (Vol. 5230, pp. 1–20). Berlin: Springer.
Rodríguez, M., Salles, F., Fabre, J.-C., & Arlat, J. (1999). MAFALDA: Microkernel assessment by fault injection and design aid. In J. Hlavička, E. Maehle & A. Pataricza (Eds.), Proceedings of the—EDCC-3—Third European Dependable Computing Conference on Dependable Computing. Lecture Notes in Computer Science (Vol. 1667, pp. 143–160). Berlin: Springer.
Shiravi, A., Shiravi, H., Tavallaee, M., & Ghorbani, A. A. (2012). Toward developing a systematic approach to generate benchmark datasets for intrusion detection. Computers and Security, 31(3), 357–374.
Shirey R. (1999). Internet Security Glossary. Internet Engineering Task Force, RFC 2828. https://tools.ietf.org/html/draft-shirey-security-glossary-01
Sinha, S., Jahanian, F., & Patel, J. M. (2006). WIND: Workload-aware INtrusion detection. In D. Zamboni & C. Kruegel (Eds.), RAID 2006—Proceedings of the 9th International Symposium on Recent Advances in Intrusion Detection, Hamburg. Lecture Notes in Computer Science (Vol. 4219, pp. 290–310). Berlin: Springer.
Sommer, R., & Paxson, V. (2010). Outside the closed world: On using machine learning for network intrusion detection. In Proceedings of the 2010 IEEE Symposium on Security and Privacy, Oakland (pp. 305–316). Washington, DC: IEEE Computer Society.
Sperotto, A., Sadre, R., Vliet, F., & Pras, A. (2009). A labeled data set for flow-based intrusion detection. In G. Nunzi, C. Scoglio & X. Li (Eds.), IPOM 2009—Proceedings of the 9th IEEE International Workshop on IP Operations and Management, Venice. Lecture Notes in Computer Science (Vol. 5843, pp. 39–50). Berlin: Springer.
Stallings, W. (2002). Cryptography and network security: Principles and practice (3rd ed.). London: Pearson Education.
Stolfo, S., Fan, W., Lee, W., Prodromidis, A., & Chan, P. (2000). Cost-based modeling for fraud and intrusion detection: Results from the JAM project. In Proceedings of the DARPA Information Survivability Conference and Exposition (DISCEX), Hilton Head, SC (Vol. 2, pp. 130–144). Piscataway, New Jersey: IEEE.
University of California (1998). KDD Cup 1999 Data: Data set used for the third international knowledge discovery and data mining tools competition. http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html
Witten, I. H., Frank, E., & Hall, M. A. (2011). Data mining: Practical machine learning tools and techniques. Morgan Kaufmann Series in Data Management Systems (3rd ed.). Burlington, MA: Morgan Kaufmann.
Wright, C., Cowan, C., Smalley S., Morris, J., & Kroah-Hartman, G. (2002). Linux security modules: General security support for the Linux Kernel. In Proceedings of the 11th USENIX Security Symposium, San Francisco (pp. 17–31). Berkeley, CA: USENIX Association.
Yu, S., & Dasgupta, D. (2011). An effective network-based Intrusion detection using conserved self pattern recognition algorithm augmented with near-deterministic detector generation. In Proceedings of the 2011 IEEE Symposium on Computational Intelligence in Cyber Security (CICS 2011), Paris (pp. 17–24). Piscataway, New Jersey: IEEE.
Zhang, Y., Wang, H., Gu, Y., & Wang, D. (2008). IDRS: Combining file-level intrusion detection with block-level data recovery based on iSCSI. In Proceedings of the Third International Conference on Availability, Reliability and Security (ARES 2008), Barcelona (pp. 630–635). Washington, DC: IEEE Computer Society.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Kounev, S., Lange, KD., Kistowski, J.v. (2020). Software and System Security. In: Systems Benchmarking. Springer, Cham. https://doi.org/10.1007/978-3-030-41705-5_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-41705-5_18
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41704-8
Online ISBN: 978-3-030-41705-5
eBook Packages: Computer ScienceComputer Science (R0)