Research On Anomaly Detection Of Program Behavior Based On Linux Security Module

Due to the prevailing of computer security problem, intrusion detection has been recognized as an important direction of research. Especially anomaly detection, which has the capability to detect unknown attacks, has been paid more and more attention to. The existing approaches to host-based anomaly detection mainly focus on utilizing system calls as data source and then modeling.On the basis of analysis to various anomaly detection methods, a new approach, which adopts the Linux kernel 2.6.9 supported LSM (Linux Security Module) data as data source, and then builds a Hidden Markov model to detect intrusion, is designed. Because LSM is specially designed for Linux security; its data extract points are located in system calls or kernel functions, analysis shows that LSM data provides more detailed information than system calls, and have more information about security. How to extract LSM sequence from application programs and then pre-process is well researched; and how to train the captured LSM sequences before detection for constructing Hidden Markov model is analyzed. A new detection algorithm is used to detect the monitored program's LSM sequences real-time. For the purpose of validating LSM data source and the detect ability, an anomaly detection prototype system-HIDS, which extracts LSM data from the monitored program and then constructs Hidden Markov model, is designed and realized. The HIDS system uses netlink and other technologies, and it is composed of 4 modules: the LSM monitor module, the kernel and the user space communication module, the HMM event analysis module, the detect record and response module. The first two modules works in kernel space, the last two works in user space.HIDS load performance, training experiment and simulation attacks are tested. The test result shows that, it's efficient that LSM is used as the anomaly detection data source. The method, which adopts LSM data under Linux and then builds a Hidden Markov Model to detect anomaly, is feasible. It produces good detection results.