Virtualization-Based Approaches for Mitigation of Malware Threats

Modern computer systems consist of a number of software layers to provide efficient resource management, secure isolation, and convenient environment for program development and execution. The hypervisor virtualizes multiple instances of hardware for guest operating systems or another layer of hypervisors with strong isolation between the virtualized hardware instances. Running directly on hardware, physical or virtualized, the operating systems (OSs) provide efficient resource management and a convenient set of interface for applications to access the hardware resource including CPU. In addition, the OS regulates each application with virtual memory, user identification, file permissions, etc. such that a non-privileged application cannot interact with other applications or access privileged files without gaining the corresponding permissions. One level above the OS, the runtime libraries and system daemons help applications to communicate with the OS in an easier manner. Each software layer has various software bugs and design flaws such that the secure isolation could be broken through. This motivates our research of securing different software layers with a series of virtualization-based approaches.;In this dissertation, we firstly present an OS-level virtualization system, AirBag, to improve Android architecture in the aspect of malware defense and analysis for dealing with flaws in Android runtime environment. AirBag allows users to "test" untrusted apps in an isolated Android runtime environment without private data leakage, system files corruption, and more severe damages such as sending SMSs to premium numbers. Besides, users can "profile" untrusted apps in the instrumented isolated Android runtime, which improves the capabilities of dynamic analysis. However, such an OS-level approach is vulnerable to attacks that exploit the vulnerabilities inside the OS. When the OS is compromised, all private data such as the bank account and password could be leaked while the amount of an online payment could be changed by the attacker. Since building a bug-free OS is impossible, we present a tiny hypervisor, tHype, to provide trusted IO access to users when they input the sensitive data or perform critical operations. Compared to existing hypervisors, tHype outperforms by its code size since it only virtualizes critical IO on mobile devices, mainly touchscreen and framebuffer. Yet, in general, virtualizing computer systems is complicated such that most existing hypervisors have large code base which make them vulnerable. Even worse, the hosted (or Type-II) hypervisor is considered to include the host OSs in its trusted code base (TCB) that makes it have a wider attack surface compared to bare-metal (or Type-I) hypervisors.We present the DeHype system to reduce the TCB of the hosted hypervisor by deprivileging its execution to user mode. With DeHype, the hypervisor is executed in the context of an user mode process for each guest VM, which prevents a compromised hypervisor from attacking other guests.