Detecting Stealthy Malware Using Behavioral Features in Network Traffic

It is clearly in the interest of network administrators to detect hosts within their networks that are infiltrated by stealthy malware. Infected hosts (also called bots) can exfiltrate sensitive data to adversaries, or lie in wait for commands from a bot-master to forward spam, launch denial-of-service attacks, or host phishing sites, for example. Unfortunately, it is difficult to detect such hosts, since their activities are subtle and do not disrupt the network.;In this thesis, we hypothesize that malware-infected hosts share characteristics in their network behaviors, which are distinct from those of benign hosts. Our approach works by aggregating "similar" network traffic involving multiple hosts. We identify key characteristics that capture basic properties of botnet operation, and that can be observed even within coarse network traffic summaries, i.e., flow records. Using network traffic collected at the edge routers of the Carnegie Mellon University campus network, and network traffic generated from real bot instances in virtual machines and honeynets running in the wild, we demonstrate that this approach can reliably detect infected hosts with very few false positives.;In addition to identifying relevant behavioral features within hosts' network activities, another contribution of this thesis is in developing efficient algorithms for analyzing network traffic. Our algorithms utilize methods from diverse areas, including statistics, data mining, machine learning, and metric embeddings. We also introduce a technique to passively infer the application implementation on a host given only anonymized traffic summaries. This technique enables us to detect malware that is browser-dependent, and can also he applied to improve the accuracy of traffic deanonymization, i.e., identifying the web sites in anonymized flow records.;To complement empirical analyses, we apply analytical models from network theory to study peer-to-peer botnets. We focus on a structural property of networks, which characterizes the tendency for edges to exist between "similar" nodes, and examine its effect on network resiliency and the network's ability to recover after a fraction of the nodes are removed. We show that previous works may have over-estimated the power of certain botnet takedown strategies, and identify an alternative strategy that is more effective than those explored previously.