A Hierarchical Network Traffic Anomaly Detection System

The rapid advancement of computer networking technologies make possible the drastic increase of both the Internet users and services being offered. To protect the legitimate users and network resources from various threats, network intrusion detection systems, together with firewalls and data encryption techniques, become the most important defense means. Among the techniques used in intrusion detection, anomaly based methods can be used to detect unknown attacks and do not need the maintenance of rule basesm. As a result, anomaly based detection is one of the current research focus in the network security field.A three-layer hierarchical model is proposed to depict the network traffic behaviors, providing different granuality for observing network traffic. First, because TCP/IP protocols poses constraints to the message exchanging process of two parties participating in a session (or pseudo-session), a seession behavior finite state machine may be used to characterize a session and measure its deviance from the specified behavior. Then, the observed metrics on individual sessions may be aggregated to measure the behaiors of a pair of communication IPs. Furthermore, these metrics of IPs can be aggregated to describe the overall behavior of the network. In this model, the information collected from the lower level is aggregated to form critical features needed to model the upper layer behavior, and the deviance between the specified behavior and the observed behavior provides an indicator to detect anomaly. It is found that many known attacks result in detectable anomaly on different layers of the proposed hierarchical model.Experiemnts are designed to verify this model, with the data set from DARPA data, Lincoln Labs, MIT. The experiment outcome shows that the proposed model is able to detect anomalies effectively and efficiently.