Design And Implemetation Of Host-Based Malcode Detection System

Host-based malicious code detection system is a intelligent system running on the host computer. It is used to detect the malicious code in the computer system and is important security software for the computer system. With society's growing dependence on computers, computer security issues become increasingly serious. Traditional anti-virus products detect malcode mainly based on signature scanning detection technology. Signature scanning technology needs to know signature byte sequence of malcode first, and then detects them by matching algorithm. An obvious drawback of this approach is that it needs for pre-built signature datebase, and therefore its capacity of new malcode detection is very weak.This article proposes methods for malcode detection based on the analysis of program behaviors. In this paper, the work and contributions can be summarized as follows:1. Analyze the principle of malcode, including kernel level rootkit, BIOS firmware malware and CPU microcode malware.2. Research methods for capture behaviors of malcode:(1) Capture the behaviors of malcode by hook technology, including a new system call interception method, drivers communications interception, etc.;search the traces of malcode by scanning, including hook code, hidden data, etc.(2) Design and implementation a method to record the malcode-related control-flow paths of program based on hardware support(Single-Step and Last-Branch Record).(3) We made innovative use of virtualization technology in the operating system to create a virtual, clean environment, so the system components which are vulnerable to malcode will work in another safe environment.(4) In order to capture the behaviors that are difficult to intercept or easy interfered by malcode, this article analyzes CPU support hardware virtualization principle, and propose a behaviors collection method based on the CPU hardware virtualization support (AMD's SVM and Intel's VMX).3. We model the operating system environment based on HMM, and using behavioral data captured by a variety of means as observed value to calculate the suspicious value of being implanted rootkit. The experiments indicate that this model performs very well in malicious rootkit code detection. At the same time, towards the dynamic control flow path data collected, we propose a method by firstly establish of a call-level tree, and then using edit-distance calculation to determine the similarity to detect the hiding malcode, the experiment has achieved good results too.4. Presided over the design of expert system based on malcode detection module, and cooperate with the members of the project team to achieve a prototype system.5. We implement a complete host-based malcode detection system by using malcode information capture module, anomaly detection algorithm module, malcode detection module based on expert system, as well as the signature scanning module as assistant module.