Research On The Detection Of Anti-Analysis Behavior In Malware

Malware is the root cause of many information security threats. Security companies and researchers develop automated tools to extract and analysis the runtime behaviors of malware samples. Unfortunately, malware is aware of these tools, and looks for evidence of emulated or virtualized analysis environments. If such evidence is found, malware samples reduce their malicious behaviors or simply crash, showing a different "personality" than when executed on real system.To solve the problem of anti-analysis malware, two kinds of approaches are proposed. One approach is to try to build transparent analysis platforms that are more difficult to detect by malware. But due to the performance overhead, these platforms are not suitable for the analysis of current high-volume malware feeds. Another approach runs malware samples in multiple analysis environments, and detect deviations in behaviors that may indicate anti-analysis. This approach also has some drawbacks, such as low accuracy, manual intervention and so on.The method used in this thesis falls into the second class. In addition, we made some modifications on existing approaches to improve accuracy of the detection. The main work in this thesis is as follows:1) We summarized varies of evasion techniques used by malicious programs. We discussed advantages and disadvantages of recent approaches to detect anti-analysis malware.2) We launched a deep research on applications of binary program analysis in the field of malware analysis, in particularly on dynamic binary slicing technology used in this thesis.3) We made some modifications on recent approaches to detecting anti-analysis behavior in malware. The approach proposed in this thesis can identify real anti-analysis behavior in malware, through eliminating unrelated differences between multiple analysis environments. A flexible algorithm is employed to compare the traces of system calls executed by malware across different analysis platforms. If a deviation exists, instruction traces are further compared using an efficient algorithm to determine whether the root cause of behavior deviation is anti-analysis or not.4) Based on the improved detection method, we designed and implemented a prototype system to detect anti-analysis behavior in malware. Experimental results have demonstrated that the approach can detect varies of evasion techniques, include detecting hardware characteristics, applications, time overhead and so on. When detecting malware without anti-analysis capability, our approach has better robustness.