Malicious Executables Detection Based On N-gram System Call Sequences

Along with the wide application of computer and Internet, the computer system and network security issues being increasingly concerned. Malicious code is undoubtedly the most harmful, so it has become the focus of research in the filed of computer system and network security.There are two main approaches for the detection of malicious code: static analysis and dynamic analysis. Static analysis consists in examining the code of programs to determine properties of the dynamic execution of these programs without running them. Dynamic analysis mainly consists in monitoring the execution of a program to detect malicious behavior. At present, the malicious code detection methods of the static and dynamic analysis both have a very wide range of applications, but these two methods have their own advantages and disadvantages. However, with the development of polymorphic and metamorphic technology, a large number of malicious code deformation tools emerge. Traditional signature-based static approach has been greatly challenged. The dynamic method usually has to execute the executables in virtual environment, this will be powerless on those codes which hidden their malicious behavior in virtual environment.In this paper, we propose a new approach for the static detection of malicious executables based on the system call sequences. Different from the traditional implementation executing the code in virtual environment to obtain the system call sequences, this system call sequences is obtained by firstly a binary code is decompiled to assembly code, and its control flow is constructed. A pruning algorithm and a translation algorithm are presented to translate the control flow graph into a program running tree. By traveling the tree all possible execution paths of system calls are extracted. After obtained system call sequences, all executables are represented by their system calls. At last three classifiers, KNN, SVM, and decision tree, are adopted to classify binary codes.Experimental results on the malicious and normal samples show that this malicious detection method has high accuracy, low false positive rate and low false negative rate, able to achieve polymorphic, metamorphic and unknown malicious detection.