Keywords

1 Introduction

A malicious software, or malware for short, is a software used or created by an attacker to perform his bad intentions on a computer system without authorisation and knowledge of its user. Stealing safety or liability-critical information and damaging the compromised system is mainly targeted. The recent developments in the field of computation systems and the proliferation of systems such as smart phones, tablet, Internet of Things (IoT), cloud computing have led to an increased interest in malware development. According to [1], more than 430 million new malware samples were released in 2015 with an increase of 36% from the previous year, and a new zero-day vulnerability was discovered at each week on average with an doubled release frequency in comparison with the previous year. Security tools compare suspicious files with their malware definition database, which are constructed based on known security issues by analysts. Based on definition database, these tools check whether a given file is malware or not. For example, anti-viruses running on end-user computers are mainly signature-based solutions. From an abstract point of view, they read the suspicious file in a binary format and look for a match with their signature database. According to [2], anti-virus solutions are fast and effective for known malware but their accuracy of analysis and adequacy are easily degraded by slightly changed malware. Although signature-based solutions are subject to delay on identifying new and obfuscated version of malware and updating their databases, they are convenient over previously registered attacks. Subject to an increasing number of sophisticated, advanced, and targeted attacks (a.k.a., advanced persistent threat, APT), security researchers and practitioners have explored more robust and timely solutions to new and unknown threats. Generally, the security community prefers sandboxing where malware activities are monitored during their execution in an isolated and controlled environment. These systems track and inform about file, registry, network, and process activities. To successfully detect malware and take appropriate counter measures, dynamic analysis can be considered as an integrated scenario and solution of an environment provided to malware for being deployed and performing its tasks. CWSandbox leverages API hooking technique in user mode to track malware’s activities [3]. Once the sample is loaded into memory, API hooking is performed by in-line code overwriting. The sample is executed in a suspend mode and then all loaded DLL’s API functions are overwritten. Hence, CWSandbox collects all called functions and their related parameters. Then, it generates a high-level report about activities and malware analyst can quickly follow them. Since it collects data in user mode, low level operations and undocumented function calls can not be captured. Cuckoo is an open source analysis system and relies on virtualization technology to run a given file [4]. It can analyze both executable and non-executable files. These activities including pre-defined Win API functions and their parameters are monitored and captured by its user-space API hooking technique. Owing to the fact that it runs at user level, malware can easily notice presence of the analysis attempt causing to change its behaviour. Capture-BAT is another dynamic analysis tool developed by New Zealand chapter of honeynet.org [5]. Capture-BAT monitors process, registry, and file activities at kernel level, and it captures network traffic using winpcap library. Furthermore, it offers selection of events through its filtering interface that can be used by the analyst to prevent noisy events to be captured. Since Capture-BAT is not an automated malware analysis system, serious concerns exist on whether it can efficiently handle the high penetration of new and existing malware. To the best of our knowledge, dynamic analysis systems still use old versions of Windows OS, for instance Cuckoo merely employs Windows XP and 7, as an underlying analysis environment. However, computer owners generally prefer to upgrade their OS to the newest version. Therefore, existing systems may fail at analysing malware targeting new versions of Windows OS. Reliability and availability of malware analysis scheme may be a major concern subject to the release of new Windows OS version. Consequently, the next generation of dynamic malware analysis solutions should be adaptable to the future versions of OS. This chapter is an extended version of [6]. In this chapter, kernel callback mechanism is integrated which allows detailed view of run-time events based on defined conditions on a system basis and new malware samples with their targeted features are elaborated and evaluation study is enriched. This chapter is organized as follows: The implementation details of VirMon components and their functionalities are elaborated in Sect. 2. In Sect. 3, two real-world malware samples captured in the wild are used to illustrate the effectiveness and analysis results about monitoring the malware activities. Some conclusions are given in Sect. 4.

Fig. 1.
figure 1

Overview of the analysis machine components (e.g., filter drivers)

Full size image

2 Design of VirMon: System Components and Functionalities

Analysis machine components include mini-filter driver and driver manager. They are responsible for reporting host-based process, registry, and file system activities performed by the analysed file, see Fig. 1. Activities of the process (or processes) initiated by that executable and (if any) child process (i.e., additional process created by some of these processes) are monitored by means of the kernel callback functions being embedded in kernel driver. API hooking is one of the preferred methods for dynamic malware analysis. We use kernel callback mechanism, which provides detailed view of run-time events based on defined conditions on a system basis [7]. To be able to use callbacks, a kernel driver needs to be built. Basically, this driver, also known as mini-filter driver, intercepts all IRP requests made by an application and decides whether allowing or refusing these operations according to the given rule set. The portability is followed by the claim of Microsoft, which states that kernel-callback mechanism is reliable and compatible with all versions of Windows including their 64 bit versions. Malware creates new process or changes an existing one in order to run its actions without being detected. To obtain the information about run-time events provided by the callback mechanism, related functions need to be called with their relevant parameters. For process monitoring, the “PsSetCreateProcessNotifyRoutine” function allows the mini-filter driver to monitor changes applied to the running processes. Many malware uses Windows registry to gain persistent access to the system. Attackers can use these registry keys to gain authority over the OS and persistent access to it. When an event is occurred on the registry, “CmRegisterCallback” function can provide related information to the mini-filter driver. To track registry events, one needs to identify which actions to be monitored in the driver via some self-explanatory constant values (e.g., RegNtPostCreateKey, RegNtPreDeleteKey, RegNtEnumerateKey). In VirMon, registry operations about OpenKey, CreateKey, DeleteKey, SetValueKey, DeleteValueKey, QueryValueKey and EnumerateKey are monitored. Malware copies itself or its variants to various locations in the file system and then adds a registry key to start automatically while booting. “FltRegisterFilter” function along with its callback actions can be used to monitor file system activities on the system. Like registry monitoring, the actions to be tracked have to be addressed in the driver accompanied by some constant values (e.g., IRP_MJ_WRITE, IRP_MJ_READ, IRP_MJ_QUERY_INFORMATION). In VirMon, to avoid redundant and distracting file operations, we consider only read, write, and delete events performed by the tracked processes. Network components are responsible for reporting network activities of the analysed file. Malware needs to connect very often to the C&C servers to send confidential information collected from compromised machines or to receive C&C servers’ commands. This bi-directional communication makes analysis of the network activities of malware as an inevitable requirement to be fulfilled by malware researchers. In VirMon, we use different network solutions, such as VLAN, VPN, IPDS, and firewall to monitor network activities of suspicious files. Intrusion Prevention and Detection System (IPDS) is a network security solution monitoring network traffic and system activities [8]. In VirMon, an IPDS is introduced to prevent possible networks attacks caused by suspicious files in the system. This secure scheme fulfils the analysis requirement of the malware analyzer by giving an opportunity to acquire all network events including the requested web pages, downloaded files by malware. Suricata, an open source IPDS solution, can prevent malicious attacks such as distributed denial of service (DDoS), port scanning and shell codes [9]. It can also extract files and HTTP requests from live network traffic. In order to circumvent network attacks caused by malware sample under dynamic analysis, we use Suricata as one of the IPDS component. Bro [10] is different from the typical IPDS since it can not block the attacks and does not rely on network signatures but it enables monitoring all network traffic. It supports well-known network protocols, extracts related information from network packets and exposes network activities at high-level [11]. In the hierarchy of VirMon, Bro runs on IPDS server and reads network interface. To extract files from live network traffic, a custom script compatible with HTTP, FTP, IRC, SMTP protocol is created. The developed script logs hostname, URL, filename, file type, and transport layer information (e.g., IPs and ports) to a file which is parsed periodically for storing these information in database. Meanwhile, if the files extracted by Bro Engine have not already been dissected beforehand, they are queued into application server’s priority queue. The open source Oracle Virtual-Box [12] is chosen as the virtualization infrastructure to host and to deploy malware analysis machines. Virtual-Box supports both 32 and 64 bit CPU. Virtual-Box also provides accessibility features such as remote machine management, display of multiple remote machines via web interface, and offers command-line interface (VBoxManage) for automated tasks. The application server is responsible for the management of the malware analysis processes. It assigns an analysis machine to the submitted file. It collects the file activities from analysis system components. Then, it formats the collected data and stores them in a database. After the analysis operations are completed, the application server commands the analysis machines to be restored to a clean state.

Fig. 2.
figure 2

Information flow and interactions between application server and analysis machine

Full size image

3 Evaluation

The procedure followed by VirMon to analyze a submitted file, and the interaction between the application server and the analysis machine is plotted in Fig. 2:

  • Analysis application sends a request to the application server to start a new analysis process.

  • The application server chooses a file in its priority queue and sends it to a analysis machine.

  • The analysis application injects codes into explorer.exe process. This process executes submitted file in the suspended mode and then explorer.exe writes PID of this recently created process to a shared memory. Subsequently, the analysis application reads PID info from the shared memory, sends a message involving this information to the driver and a request about recording the events initiated by this process.

  • The analysis application waits until the analysed process exits or a timeout of 3 min occurs. Then, it sends a message to the driver to stop recording events.

  • The driver stops recording and writes collected events to a log file.

  • The analysis application sends the log file to the application server for parsing.

  • The application server parses the log file and stores it in the database. Finally, it reverts the analysis machine to the clean state.

We analyze a trojan, named as Hesperbot [13] detected on August, 2014. This trojan is focused on stealing banking account information to be used towards unauthorized money transfers. The attackers social engineer victims to execute attached files by sending e-mail which looks like it is originated from one of the service providers in Turkey. The analysis of hesperbot is executed automatically on a Windows 8 Ultimate 64-bit OS. Since the number of events including system dll file and registry accesses gathered from VirMon for this malware is too high (10000+), only important events occurred on the system are displayed in Table 1. The intention of the malware sample can be easily derived from this table. When the sample is executed, the process created by explorer.exe creates a new process entitled “fatura_874217.exe” having the same name in its directory. In Turkish, “fatura” means “the bill”. This technique, named as process hollowing, [14, 15], has been recently used by malware to hide itself. Then, the process created by explorer.exe terminates itself. The fatura_874217.exe process created by hollowing technique, creates %APPDATA%\Sun and %APPDATA%\yseszpkf directories and drop randomly named binary files under them. To be hidden, the fatura_874217.exe process creates new explorer.exe, which in turn is used to carry out remaining activities, download configuration files from C&C server, drop new executable and writes it to the auto-start line in the registry, respectively. This analysis shows that the VirMon dynamic malware analysis system successfully collects the run-time behaviours of the file sample.

Table 1. Important run-time activities of a trojan
Full size table
Table 2. Run-time activities of the cyrptolocker on Windows 10
Full size table

3.1 VirMon Compatibility on Windows 10

In order to show that VirMon (e.g., its mini-filter driver) is adaptable to the newest version of Windows OS, we conducted functional testing of developed mini-filter driver on Windows 10. This test is based on the analysis results of the recent malware sample, known as cyrptolocker, a variant of ransomware, that encrypts sensitive documents on the infected machine and forces to pay a ransom to make them usable again. Since other system components of VirMon work independently, it has been sufficient to install the developed mini-filter driver into the analysis machine to integrate new OS. Accordingly, we successfully installed VirMon’s mini-filter driver on Windows 10 without any need to modify or build driver’s code. Table 2 shows the summarised run-time activities of the cyrptolocker still observed by VirMon during its analysis. Cyrptolocker malware sample (see [16]) was active in the wild and a signature has not yet been created at the time of writing this paper. This malware’s activities show that it uses the process hollowing technique as in the previously analyzed sample (i.e., hesperbot sample). Then, it probably searches specific file types under C:\drive to encrypt and makes them unusable unless one does not have the description key. It sends some information to its C&C server and asks ransom from its victim user.

4 Conclusions

In this paper, the virtualization-based dynamic malware analysis system and its components is presented. A mini-filter driver is built to monitor run-time activities of the file to be analyzed. Since kernel-callback scheme is reliable and compatible with all versions of Windows OS, analysis machine can support all Windows OS versions. In addition to its portability feature, the design supports virtualization and scalability, for instance the average rate of analysis can be adjusted by the virtualization approach. Two recent malware samples captured in the wild are analyzed to illustrate the portability feature and analysis success of the presented dynamic analysis system. The activities of the analyzed samples are extracted accurately and details of each activity are given with the timestamp, event and process description, which enhances readability of the analysis.