Study On Program-Based Anomaly Detection

A program-based anomaly detection approach is discussed, which takes both advantage of the ability of anomaly detection in detecting novel attacks and the stability of program behavior in intrusion analysis compared with other observables. Two new such methods are proposed. One is based on the neural network, which uses a recurrent backpropagation neural network to identify anomalies by predicting future patterns of program execution traces. The other is based on the hidden Markov model (HMM), named as HMMTide method, which detects anomalies by matching the local pattern of HMM states sequences generated by new behaviors with the established normal model. Both two methods have the ability of generalizing from incomplete data with less requirements of system resource such as storage space. Especially, the recurrent neural network method is effective in improving detecting rate, while the HMMTide method is effective in reducing false positives. The two methods are both tested on the data provided by University of New Mexico. The results of our preliminary experiments have shown that both methods have improved the property of intrusion detection system. In addition, a new idea of anomaly detection integrating the neural network with the HMM is also presented.