Research On ROP Attack Detection Method Based On Dynamic Feature Monitoring

The security of operating systems has always been a major concern,and ReturnOriented Programming（ROP）is one of the most dangerous attacks on the security of all operating systems.The ROP searches for a sequence of commands from the vulnerable program that ends with a RET command and has some functionality,and effectively links them into a gadget chain.The ROP then hijacks the program execution stream to this gadget chain,thus enabling the attack.Many detection and defense techniques have been proposed for ROP,such as stack detection,control flow integrity and address randomization,but all of these detection and defense techniques have limitations in terms of accuracy or efficiency.In addition,advances in defense technology have led to the emergence of ROP variants,which are more widely exploited,more versatile and have more covert behavioral characteristics,and can bypass most existing detection and defense techniques.In this thesis,we analyze the dynamic characteristics of attacks and propose a detection method,ROPGMN,which determines whether an attack has occurred by calculating the similarity between the current execution flow and the normal execution flow.In this thesis,we also investigate ROPMFilter,a fused multidimensional execution stream filtering method,in order to reduce the performance overhead of ROPGMN.This method only feeds execution streams suspected to have attack characteristics into ROPGMN for detection,greatly improving detection efficiency.The main work in this thesis is as follows:First,in this thesis,we study the statistical and structural properties of the variant gadget chain during an attack.And we propose four detection strategies for the characteristics of the links between the gadget’s entry and exit addresses and the gadget.The four detection strategies are Instruction balancing,variant gadget thresholds,return address checksum and function pointer checksum.We propose an efficient execution flow filtering method,ROPMFilter,which can effectively filter out execution flows suspected of having attack characteristics.Secondly,we propose an attack detection method ROPGMN with high accuracy in combination with graph neural networks.This method compares the program execution streams filtered by ROPMFilter with the normal execution streams prepared prior to detection by graph matching and derives a similarity score.If the similarity score is lower than the set similarity threshold,the ROP and its variants are judged to exist.Because the graph extraction process and the feature extraction process fully consider the characteristics of the variant gadget,ROPGMN can cover the features of the attack chain very well.Its judgment results possess a high accuracy rate.The overall detection technique is highly efficient again because ROPMFilter filters the execution stream and greatly reduces the pressure of graph matching.Finally,in this thesis,we have designed and implemented a prototype detection system for ROP and its variants on 64-bit Linux systems based on ROPMFilter and ROPGMN.Numerous experiments on ROP and its variants attacks were conducted in this system to evaluate the differences between the detection tool proposed in this thesis and other ROP detection means in several aspects.Numerous experiments on ROP and its variants attacks were conducted in this system to evaluate the differences between the detection tool proposed in this thesis and other ROP detection means in several aspects.The comparative metrics are accuracy,efficiency and coverage.The accuracy rate reached 98.73%,the false alarm rate was as low as 0.11%,and the coverage rate was much higher than other tools,which can effectively detect a wide range of ROP variants.