Research On Code Reuse Attack Protection Technique Based On Virtual Machine Monitor

With the rapid development of information technology,Cyberspace is revolutionizing the human’s production and lifestyle.At the same time,computer and network attacks are emerging in an endless stream.Cyberspace security is facing serious risks and challenges.The code reuse attack(CRA)has become one of the mainstream attacks recently.Unlike traditional viruses and Trojans,CRA doesn’t need to introduce additional malicious code from the outside,instead it leverages the existing code of the program to implement a malicious attack.As the CRA becomes more and more mature,it derives out a lot of new varieties and poses a huge challenge to the existing code reuse attack detection and defense technology.Aiming at preventing the CRA,this thesis analyzes the attack principle of both the traditional CRA and new CRA in depth,then puts forward two kinds of protection methods and carries out indepth studies to make up for the deficiencies of existing CRA defense methods.The major works and innovations of this dissertation are as follows.(1)Existing runtime randomization methods have a coarse-grained size and can not avoid TOC-TOU attacks.According to the deficiencies of existing runtime randomization methods,a runtime rerandomization based method of preventing CRA is proposed.Firstly,the input syscalls of the target process are monitored in VMM.Secondly,based on the running state of the target process,it will trigger a runtime rerandomization for the target process and reorder the basic blocks in the function.Then,the target process will become a moving target and prevent attackers from exploiting information leaks to get the code in memory.A test platform is implemented based on the LLVM compiler and virtualization technique.The validity and the performance overhead introduced by Mixer are verified and analyzed.The test results show that this method can protect against the CRA effectively with less than 20.3% performance penalty.(2)The protection method against CRA based on the discete TLB structure relies on special hardware structure and lacks universality.According to this deficiency,we propose a CRA protection technique VXnR based on code anti-leakage.In this method,we set Execute-no-Read(XnR)permission for the code pages of the target and prevente information leaks,which can defense CRA.The VXnR method is implemented in VMM and does not need to modify the client operating system kernel,so it has good compatibility and security.The effectiveness and performance overhead in the VXnR method are verified and analyzed.The test results show that VXnR can prevent the CRA with less than 52.1% overhead.(3)We design and implement a VMM-based CRA defense prototype system.On this basis,we verify the validity of the method of combination of the runtime randomization and the code anti-leakage technique.We also test and analyze the performance penalty introduced by HVMdefender.The test results show that HVMdefender can effectively defense both the traditional CRAs and new CRAs within acceptable performance range.