Abstract
Contemporary malware makes extensive use of different techniques such as packing, code obfuscation, polymorphism, and metamorphism, to evade signature-based detection. Traditional signature-based detection technique is hard to catch up with latest malware or unknown malware. Behavior-based detection models are being investigated as a new methodology to defeat malware. This kind of approaches typically relies on system call sequences/graphs to model a malicious specification/pattern. In this paper, we present a new class of attacks, namely “shadow attacks”, to evade current behavior-based malware detectors by partitioning one piece of malware into multiple “shadow processes”. None of the shadow processes contains a recognizable malicious behavior specification known to single-process-based malware detectors, yet those shadow processes as an ensemble can still fulfill the original malicious functionality. To demonstrate the feasibility of this attack, we have developed a compiler-level prototype tool, AutoShadow, to automatically generate shadow-process version of malware given the source code of original malware. Our preliminary result has demonstrated the effectiveness of shadow attacks in evading several behavior-based malware analysis/detection solutions in real world. With the increasing adoption of multi-core computers and multi-process programs, malware writers may exploit more such shadow attacks in the future. We hope our preliminary study can foster more discussion and research to improve current generation of behavior-based malware detectors to address this great potential threat before it becomes a security problem of the epidemic proportions.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., Mitchell, J. C.: A Layered Architecture for Detecting Malicious Behaviors. In: Proceedings of the 11th international Symposium on Recent Advances in intrusion Detection (RAID’08) (2008)
Lattner, C., Adve, V.: LLVM: A compilation framework for lifelong program analysis & transformation. In: Proceedings of the 2004 International Symposium on Code Generation and Optimization (CGO’04) (2004)
Christodorescu, M., Jha, S., Seshia, S., Song, D., Bryant, R.: Semantics-Aware Malware Detection. In: Proceedings of IEEE Symposium on Security and Privacy (2005)
Barford, P., Yagneswaran, V.: An Inside Look at Botnets. In: Advances in Information Security. Springer, Berlin (2006)
Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th ACM conference on Computer and communications security (CCS’02) (2002)
Filiol E.: Formalisation and implementation aspects of k-ary (malicious) codes. J. Comput. Virol. 3(3), 75–86 (2007) (EICAR 2007 Best Academic Papers)
Harbour, N.: Stealth Secrets of the Malware Ninjas. https://www.blackhat.com/presentations/bh-usa-07/Harbour/Presentation/bh-usa-07-harbour.pdf.
Kolbitsch, C., Comparetti, P.M., Kruegel, C., Kirda, E., Zhou, X., Wang, X.: Effective and Efficient Malware Detection at the End Host. In: Proceedings of 18th USENIX Security Symposium (2009)
Nomenumbra: Counter Behavior Based Malware Analysis, Hacking at Random. HAR (2009)
Aciiçmez, O., Koç, Ç.K., Seifert, J.: On the power of simple branch prediction analysis. In: Proceedings of the 2nd ACM Symposium on information, Computer and Communications Security (ASIACCS’07) (2007)
Kernighan B.W., Lin S.: An efficient heuristic procedure for partition graphs. Bell Syst. Tech. J. 49, 291–307 (1970)
Christodorescu, M., Jha, S., Kruegel, C.: Mining specifications of malicious behavior. In: Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (2007)
Anubis. http://anubis.iseclab.org/
Lamport L.: Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21(7), 558–565 (1978)
Jiang, X., Walters, A., Buchholz, F., Xu, D., Wang, Y.M., Spafford, E.H.: Provenance-Aware Tracing of Worm Break-in and Contaminations: A Process Coloring Approach. In: Proceedings of 26th IEEE Int’l Conf. Distributed Computing Systems (ICDCS’06) (2006)
Fletcher, T.: Sharing a File Descriptor Between Processes. http://www.qnx.com/developers/articles/article_913_1.html
Yin, H., Song, D., Manuel, E., Kruegel, C., Kirda, E.: Panorama: Capturing system-wide information flow for malware detection and analysis. In: Proceedings of the 14th ACM Conferences on Computer and Communication Security (2007)
King, S.T., Chen, P.M.: Backtracking Intrusions. In: Proceedings of the 2003 Symposium on Operating Systems Principles, pp. 223–236 (2003)
Kirda, E., Kruegel, C., Banks, G., Vigna, G., Kemmerer, R.: Behavior-based Spyware Detection. In: Proceedings of the USENIX Security Symposium (2006)
Cohen F.: Computer viruses: theory and experiments. Comput. Secur. 6(1), 22–35 (1987)
Phoenix. https://connect.microsoft.com/Phoenix
Cavallaro, L., Saxena, P., Sekar, R.: On the limits of information flow techniques for malware analysis and containment. In: Proceedings of 5th international conference on Detection of Intrusions and Malware, and Vulnerability Assessment (2008)
Szor P.: The Art of Computer Virus Research and Defense. Addison-Wesley Professional, Reading (2005)
Forrest, S., Perelson, A.S., Allen, L., Cherukuri, R., Self-Nonself Discrimination in a Computer. In: Proceedings of IEEE Symposium on Security & Privacy (1994)
Stinson, E., Mitchell, J.C.: Characterizing Bots’ Remote Control Behavior. In: Detection of Intrusions & Malware, and Vulnerability Assessment (2007)
Willems, C., Holz, T., Freiling, F.: Toward Automated Dynamic Malware Analysis Using CWSandbox. In: Proceedings of IEEE Security and Privacy (2007)
Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Automating mimicry attacks using static binary analysis. In: Proceedings of the 14th conference on USENIX Security Symposium (2005)
Norman Sandbox Whitepaper. http://www.norman.com
Srivastava, A., Lanzi, A., Giffin, J.: System Call API Obfuscation. In: Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection (2008)
Rieck, K., Holz, T., Willems, C., Düssel, P., Laskov, P.: Learning and Classification of Malware Behavior. In: Proceedings of Detection of Intrusions and Malware, and Vulnerability Assessment (2008)
Percival, C.: Cache missing for fun and profit. BSDCan (2005). http://www.daemonology.net/hyperthreading-considered-harmful/
Stevens R.: UNIX Network Programming, 2nd edn. Interprocess Communications, vol. 2. Prentice Hall, Englewood Cliffs (1999)
Dyshlevoi, K.V., Kamensky, V.E., Solovskaya, L.B.: Marshalling In Distributed Systems: Two Approaches (1997). http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.26.9781
Borello J., Mé L.: Code obfuscation techniques for metamorphic viruses. J. Comput. Virol. 4, 211–220 (2008). doi:10.1007/s11416-008-0084-2
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ma, W., Duan, P., Liu, S. et al. Shadow attacks: automatically evading system-call-behavior based malware detection. J Comput Virol 8, 1–13 (2012). https://doi.org/10.1007/s11416-011-0157-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11416-011-0157-5