Software Behavior Analysis Method Base On Behavior Template

With the development of information technology, the software has met the various needs of people. However, as the size of software increases gradually, it becomes more vulnerable. While abnormal software threatens the normal operation of computer any time. It leads to more challenge in software security.In the response to this challenge, we propose software behavior analysis system based on the behavior template. Our method applies the combination of static and dynamic analysis method to establish model of software behavior, which can detect the abnormal behavior. The system consists of three modules including preprocessing module, modeling module and detection module.First, software interrupt is applied in pre-processing module, which set between the functions by the debugger attach. Software interrupt is access to correspondence between the function and system call sequences in modeling module.Secondly, a new bilayer software behavior map is proposed. The bilayer software behavior map includes function transfer map and minimum function block transfer map. We built the automata based on behavior template with function block transfer map as learning set. Function transfer map utilize the method of control flow analysis, and minimum function block transfer map is built by analyzing sources code marked. We build the finite state automaton which can be used to describe the software behavior more completely and detect abnormal behaviors automatically finally.Last, the finite state automaton based on behavior template detects test software. Each state in automaton is matched by system call sequence called by test software. We apply two kinds of detection method, one is coverage testing of module, and another is detection method of variable-length sequences comparison we proposed. The method converted the variable-length sequences to fixed-length sequence, calculate the distance between sequences and determine the similarity. This method breaks the limitation of fixed-length sequence in similarity test.In addition, the methods applied in Windows Linux and the platform in experiment. On the Windows, we realize the modeling module, variable-length software sequence detection methods though the RSS reader as instance. Meanwhile, on the Linux platform, our method, N-gram, Var-gram and the FSA model carries on the contrast experiment to detect the vulnerabilities in Bash 4.3 and Tcpdump. Experiments proved that our method can detect the attack behavior effectively with smaller size of the model files.