How to detect the Cuckoo Sandbox and to Strengthen it?

Abstract

Nowadays a lot of malware are analyzed with virtual machines. The Cuckoo sandbox (Cuckoo DevTeam: Cuckoo sandbox. http://www.cuckoosandbox.org, 2013) offers the possibility to log every actions performed by the malware on the virtual machine. To protect themselves and to evande detection, malware need to detect whether they are in an emulated environment or in a real one. With a few modifications and tricks on Cuckoo and the virtual machine we can try to prevent malware to detect that they are under analyze, or at least make it harder. It is not necessary to apply all the modifications, because it may produce a significant overhead and if malware checks his execution time, it may detect an anomaly and consider that it is running in a virtual machine. The present paper will show how a malware can detect the Cuckoo sandbox and how we can counter that.

1 Introduction

The Cuckoo Sandbox [1] is an open source malware analysis system. The development has started in summer 2010 in a Google Summer Code project. The actual version is 1.0. Its goal is to provide a way to analyze files automatically and to provide all the interactions between the files under analysis and the system. The main targets are Windows executables, DLL, Internet URLs, PDF, Office documents and Java files. In this paper, we present how a malware can detect whether Cuckoo is trying to analyze it or not. For each attack presented, we give the mechanism used for the detection of Cuckoo and an example of code which allows to realize that. The presented techniques are working on the 0.4 and 0.5 versions of Cuckoo. Because it is a very active project, future modifications may integrate further protections to prevent our attacks which are presented in this paper.

We will speak about solutions to avoid the detection. In some cases, the same countermeasure can be used to counter two or three attacks, so we give the most specific solution in priority.

Finally, we will discuss about the detection of the virtual machine which is hosted by Cuckoo. Actually a good proportion of malware try to detect whether they are in a virtual machine or a physical machine. With the knowledge of internal mechanisms in Cuckoo, it is possible to add some code in order to fool the malware by giving erroneous information.

2 Detection of Cuckoo

In this part, we present the different techniques that may be used by malware to detect whether they are under Cuckoo analysis or not.

2.1 Hooks’ detection

The analysis of the dynamic link library cuckoomon.dll source code and particulary the files cuckoomon.c and hooking.c provides information about the technical implementation of hooks. Currently the only technique used is HOOK_JMP_DIRECT, as shown in the following code:

The function, for this type of hook, which is defined in the file hooking.c, is sufficiently explicit about the implementation method as shown in the following extracted code:

With this information and the list of the hooked functions, given by the table hook_t g_hooks[] in the file cuckoomon.c, it is easy to create a code whose purpose is to obtain the address of one function and check its first opcode.

2.2 Folder’s detection

By default, Cuckoo uses a specific folder on the guest system in order to store and retrieve some information to the host. Under a Windows virtual machine, the default directory is c: $\big \backslash$ cuckoo.

Indeed it is relatively easy to detect it by looking for the presence of the folder with a possible code like the following one:

2.3 Pipe’s detection

In a similar manner we used for Cuckoo’s directory, it is possible to detect the presence of the pipe used to communicate between the host system and the guest system. Since the name is hardcoded, it is very easy to create a small piece of code in order to detect the Cuckoo’s communication pipe.

2.4 Cuckoo’s agent detection

Even if Python is a common programming language, it is relatively rare to find it running on a Windows client subsystem. From that, in order to prevent a detection, a few malware may try to detect the running process python.exe or pythonw.exe. In case of doubt, the malware might try to detect the presence of an installation of python package on the machine or checking a few names of directories or registry keys.

2.5 Antihooking

This trick is not a real detection one, but it is more close to a way to avoid the analysis by Cuckoo. By default Cuckoo sets 3 hooks for the creation of a new process and use them to analyse the new process. One technique consists in using two executables, the first will initially restore its API calls, then run the second. As the 3 API are restored, Cuckoo will not detect that the second executable has been launched, so this one will not be analyzed. The following code shows how to restore the original API for a Windows XP with SP2/SP3 operating system.

3 Detection of VirtualBox

Because Cuckoo is in a Virtual Machine, it is important to secure the Virtual Machine to avoid the most current detection techniques. In this part, we will discuss about two sections: the first is about the detection of VirtualBox without the Guest Additions while the second is with the Guest Additions. The guest additions are used to provide closer integration between host and guest and to improve the interactive performance of guest systems. Even if they are useful for a classical use of a virtual machine, in case of malware analysis, they generally offer an advantage to the malware.

3.1 Case without the Guest Additions installed

The first possibility is to read a few registry keys. The following codes are the main used by the malware to detect a VirtualBox guest. They read the APCI, IDE and SYSTEM keys and their subkeys in order to find relationships with VirtualBox, generally with the name vbox or virtualbox.

The second technique consists in looking for the shared folders having a specific name such as VirtualBox Shared Folders. It can be done with the following code:

3.2 Case with the Guest Additions installed

In this subsection, we discuss the detection techniques of VirtualBox using the guest additions. The first one deals with finding the VBoxMiniRdrDN driver. This one is used to create shared folders between the host and the guest machine.

We can detect although the presence of a few registry keys and the Dynamic Link Library VBoxHook.dll which is used to load the different drivers for the Guest Addition.

In the spirit of the previous method we used on Cuckoo, we can detect the presence of a pipe with the specific name of the system tray which has been created by the add-on for the guest.

Another way consists in detecting any GUI process with a specific name with respect to the VirtualBox tray.

4 Countermeasures

In this section, we will speak about the different countermeasures which can be used in order to perform more stealth analysis, without being detected by malware. The first subsection relates to Cuckoo while the second deals with VirtualBox. A few of these countermeasures can be used with Cuckoo and VirtualBox.

4.1 Cuckoo

According to AlienVault  [3], it is possible to modify the cuckoomon.dll file. But this way may have performance impacts for a deep analysis because we have to check each request towards the registry table. Indeed if we try to analyze a process which manipulates a huge amount of registry keys, Cuckoo checks all the keys and compare those keys with different values. The same issue remains unchanged for files, even if they are less used.

4.1.1 Using cuckoomon.dll

To avoid some accesses to special keys, it is possible to edit the file hook_file.c and to add directly some specific response that we want to see returned to the malware. For example, we can modify the following code:

into:

Whenever the malware try to access to the subkey VirtualBox or VBox with the RegOpenKeyExA API, Cuckoo will log that and will return that the key does not exist. In the same way, we have to modify the RegQueryValueExA hook, in order to block access to some keys.

By default, Cuckoo does not log the GetFileAttributesA API, so we must add it to the source file. Using the msdn documentation [2] we can write the following code in order to log and bypass detection mechanisms from the malware:

Technically it is easy to add new hooks. For any new technique we can create a new countermeasure by adding a hook in the dll. But it takes time to react to each new attack, mainly when it depends from the number of operations performed by malicious codes. In addition, if the malware compute execution time, it may guess that it is under analysis. The quantity of logs can be too huge to allow a simple analysis.

4.1.2 Modifying Cuckoo directly

Another approach consists in modifying directly Cuckoo. Since Cuckoo is an opensource project, we can modify the agent.py file in order to change the name of the directory used for the analysis. We can use a more common name such as “windows_” or “nt” or anything else we want. So the malware will not be able to guess whether it is running with Cuckoo or not, because it will not be able to find fixed file/registry names.

For the pipe, the same system is used, but this time we have to modify the core of Cuckoo although.

The last modification that we can perform on Cuckoo is to compile the file agent.py, in order to produce an executable. We can use py2exe to perform this action. It relies on the fact that there are not real reason to find the processes python.exe or pythonw.exe running in a classical computer. So if the agent is in the form of an executable, the malware will see another process like or among many others. By doing that, the python package is no longer necessary on the guest machine.

4.2 VirtualBox

Because strengthening Cuckoo is not the only possible solution, we have to modify our virtual machines although. Some of the previous actions can be done directly with the VirtualBox Manager [4]. One of the first thing to do is to anonymize the hardware on the virtual machine. The best way to do this, is to not install the VirtualTools on the guest machine because it creates a lot of files on the system and many of them can be detected easily. The rest of the work can be divided in two parts.

4.2.1 Registry

Virtualbox creates a lot of registry keys. Those keys can be removed easily with a small batch file. Before removing those keys, it is better to copy them under another name without any reference to VirtualBox. Some keys can only be removed in the safe boot mode of Windows or require the system permission rights. The file can be as the following one:

The key names may depend on the operating system or the version number of VirtualBox.

4.2.2 Hardware

Since the version 4.2 of VirtualBox, it is possible to modify some hardware part with the VirtualBox Manager. The hdparm command give us enough information to configure correctly the hard drive and the CDROM drive.

We can change some information of the hard drive and the CDROM drive using the information from the previous command.

By default the MAC address is directly tagged as an Oracle card. In order to prevent a detection by the analysis of the network card by a malware, we can change its prefix by an other constructor.

The real information of the computer can be obtained with the command : dmidecode. The following extract shows which information can be changed:

A good idea is to change the original BIOS code with the real BIOS code of the host computer. With this modification the behavior will be really close to a real machine hosting.

5 Conclusion

In this document, we have shown how to detect Cuckoo and how we can prevent its detection by strengthening Cuckoo. More generally we can configure a virtual machine with VirtualBox in a closer way. In all cases, Cuckoo remains reliable and advanced enough to perform automatic and relatively comprehensive analyses of malware. With this hardening it becomes more difficult for malware to bypass a virtual machine equipped with the Cuckoo analysis system. The main problem is to find the best trade-off between performance and the time used to carry out such an analysis.