Abstract
Anti-debugging techniques are broadly used by malware authors to prevent security researchers from reversing engineering their created malware samples. However, the countermeasures to identify anti-debugging code patterns are insufficient, and mainly manual, which is an expensive, time-consuming, and error-prone process. There are no automatic approaches which can be used to detect anti-debugging code patterns in malware samples effectively. In this paper, we present an approach, based on instruction traces derived from dynamic malware analysis and an instruction-based pattern matching method, to detect anti-debugging tricks automatically. We evaluate this approach with a large number of malware samples collected in the wild. The experience shows that our proposed approach is effective and about 40% of malware samples in our experimental data set has been embedded anti-debugging code.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
References
Egele, M., Scholte, T., Kirda, E., Kruegel, C.: A Survey on Automated Dynamic Malware Analysis Techniques and Tools. J. ACM Computing Surveys, 1–49 (2010)
Internet Security Threat Report, vol. 16. Symantec Corporation (January 2012), http://www.symantec.com/business/threatreport/
Moser, A., Kruegel, C., Kirda, E.: Exploring Multiple Execution Paths for Malware Analysis. In: IEEE Symposium on Security and Privacy, Oakland, pp. 231–245 (2007)
Sreedhar, V.C., Gao, G.R., Lee, Y.F.: Identifying loops using DJ graphs (1995)
Yuschuk, O.: OllyDbg
Bellard, F.: Qemu: A Fast and Portable Dynamic Translator. In: The USENIX Annual Technical Conference (2005)
Chen, X., Andersen, J., Mao, Z., Bailey, M., Nazario, J.: Towards an Understanding of Anti-virtualization and Anti-debugging Behavior in Modern Malware. In: IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN 2008), pp. 177–186 (2008)
Kawakoya, Y., Iwamura, M., Itoh, M.: Memory Behavior-Based Automatic Malware Unpacking in Stealth Debugging Environment. In: Proceeding of the 5th International Conference on Malicious and Unwanted Software (2010)
Santos, I., Ugarte-Pedrero, X., Sanz, B.: Collective Classification for Packed Executable Identification. In: Proceedings of the 8th Annual Collaboration, Electronic Messaging, AntiAbuse and Spam Conference (CEAS 2011), pp. 231–238 (2011)
Yoann Guillot, A.G.: Automatic Binary Deobfuscation (2009)
Wei, T., Mao, J., Zou, W., Chen, Y.: A New Algorithm for Identifying Loops in Decompilation. In: Riis Nielson, H., Filé, G. (eds.) SAS 2007. LNCS, vol. 4634, pp. 170–183. Springer, Heidelberg (2007)
Christodorescu, M., Kinder, J., Jha, S., Katzenbeisser, S., Veith, H.: Malware Normalization. Tech. Report, No.1539, University of Wisconsin, Madison, Wisconsin, USA (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Xie, P., Lu, X., Wang, Y., Su, J., Li, M. (2013). An Automatic Approach to Detect Anti-debugging in Malware Analysis. In: Yuan, Y., Wu, X., Lu, Y. (eds) Trustworthy Computing and Services. ISCTCS 2012. Communications in Computer and Information Science, vol 320. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-35795-4_55
Download citation
DOI: https://doi.org/10.1007/978-3-642-35795-4_55
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-35794-7
Online ISBN: 978-3-642-35795-4
eBook Packages: Computer ScienceComputer Science (R0)