Research And Implementation Of Critical Techniques In Malware Detection Based On Decompilation

The broad spread of malware has presented a serious threat to the safety of computer systems. Many researchers are eager to develop a more practical and efficient detection technique to weaken the threat from malware. However, a great many of techniques, such as obfuscation, have been used by malware writers to evade current detection approaches. Reverse analysis based on decompilation is a technique to understand and analyze binary codes, by which the behavior of malware can be concluded through static analysis. It has superiority to detect the hidden threat of malware. So it is significant to investigate the malware detection based on decompilation.Based on the analysis and understanding of obfuscation technique used by malware, this thesis presents a deep study on the techniques and approaches adopted in decompilation to detect malware. First, according to the process of decompilation, classify the obfuscation used by malware into two types, pay more attention on two cases, i.e. junk insertion and subroutine exception return. Several approaches, such as virtual stack, re-decoding, control flow gap scan, are adopted to design and implement the disassemble frame and algorithms aiming at malware. Second, according to the analysis of assemble instruction sequence and the characteristics of the storages of library function name, a method to identify library function calling is proposed to deal with searching and loading library function dynamically by malware. After that, a malware detection approach based on sub-graph matching is introduced here, which identifies suspicious behavior in the executable and estimates the malicious degree through comparing control flow graph with malicious behavior defined in the malicious behavior library.The techniques and algorithms presented above have been applied to RADUX(Reverse Analysis for Detecting Unsafe eXecutables), a prototype for malware analysis and detection which is funded by the National High-Tech Research and Development Plan of China. The prototype is tested by two testing centers and used by an organization. Comparing the experimental data produced by RADUX with those by other common reverse analysis tools and famous antivirus(AV) tools, results show that the approaches introduced in this thesis, i.e., the disassemble algorithms, the methods to identify the library function and detection based on control flow graph, are favorable in their feasibility, efficiency and validity.