Capturing and analyzing Internet worms

This document is about malware analysis, with a particular focus on exploit-based Internet worms that spread from one host to another over the network by exploiting a software vulnerability in the new host being attacked. Based on our experiences analyzing real worms that use this method of worm propagation we develop a model that divides this attack into three stages: the exploit vector (epsilon) where the machine being attacked is still running its vulnerable code, the bogus control data (gamma) that is the part of the attack that is directly involved in control flow hijacking, and the payload (pi) where the worm code is being executed instead of the code of the attacked system.; The Epsilon-Gamma-Pi model will be defined more formally in Chapter 3. In this document the particular focus will be on control data attacks, but the model generalizes to hijacking of control flow at any level of abstraction. What we will show in this dissertation is that malware analysis put into the context of the Epsilon-Gamma-Pi model can take advantage of various limitations placed on the worm at each of the stages. Researchers and malware analysis professionals can benefit greatly from an understanding of the differences between the stages in terms of the adversarial model, the polymorphic and metamorphic techniques to evade signature detection, and the amount of information about the threat that can be discovered in a particular stage. Three specific examples are described in detail: Minos, an architectural mechanism to catch control data attacks in the gamma stage; DACODA, a tool to analyze attack invariants that limit polymorphism in the epsilon stage; and Temporal Search, a method to analyze the pi stage and discover timebomb attacks in a worm's payload.