Research On Intelligent Greybox Fuzzing For Performance Optimization

Fuzzing is one of the most effective approaches for testing software and identify-ing security vulnerabilities.The traditional fuzzing technique is divided into three technologies:blackbox fuzzing,greybox fuzzing and whitebox fuzzing.With the increasing of software scale,the demand for testing efficiency also increases.There-fore,researchers have focused on the intelligent greybox fuzzing with high efficiency and strong applicability.However,the current basic theory of the current greybox fuzzing is not perfect enough,so that the researchers are not profound about the essence of the testing process of coverage-based greybox fuzzing.Moreover,there are few research aiming to improve the scheduling algorithm of greybox fuzzing.Most researchers focus on improving the efficiency of mutation strategies.However,many of the recently proposed efficient techniques,which are not based on exist-ing frameworks,are not with strong applicabilities.For the work aiming for the optimization of existing techniques,though their applicabilities are strong,there are many limitations,such as the dependence of the existing inefficient mutation strategies.For the above challenges,this paper proposes a variant of the Adversarial Multi-Armed Bandit model for modeling the scheduling algorithm of coverage-based grey-box fuzzing and proposes the reward probability.Based on this model,according to the state classification model,this paper proposes a new search strategy and en-ergy schedule and implements these approaches in an adaptive energy-saving grey-box fuzzer oriented to scheduling algorithm optimization,called EcoFuzz.EcoFuzz achieves 214%of the path coverage of AFL with reducing 32%test case genera-tion of AFL.Moreover,this paper proposes a new format generation theory and a format-based path transition model and implements a greybox fuzzer with knowl-edge enhancement for mutation strategy optimization,called LearnAFL.Compared to the existing techniques,LearnAFL performs significantly better in terms of im-proving branch coverage,exploring deep paths and vulnerabilities.