Vulnerability Discovery For Encrypted Protocol Based On Fuzz-testing

21century is an era of information and networking, more and more information isexchanged through the Internet, Software vulnerabilities are increasing, and the damagecaused by these vulnerabilities is very serious. To protect the information security, moreand more software are going to encrypt their protocols. Malware writers also use encryptionto protect themselves from being reversed by security researchers. Reverse engineering onencrypted protocols can contribute to software analysis, it is also the base of vulnerabilitydigging. But now, work on reverse engineering of encrypted protocols is not perfect, thereis also no good way to discover vulnerabilities on encrypted protocols, so there is still a lotof work to do on these aspects.The first step to reverse the format of encrypted protocols is to find the decryptionfunctions, and then find the decrypted message, the last step is to analyze the protocolformat. Existing work on finding decryption functions have a lot of disadvantages, for thesedisadvantages we improved the method of finding decryption functions. On the one handwe used taint analysis and combined the function fragments belonging to the same function,on the other hand we used two new methods “relevance” and “calculate strength” to furtherfilter out decryption functions.“relevance” is used to filter the functions who have onlyhandled only part of the received encrypted messages.“calculate strength” is used to filterthe functions whose computational complexity is low. On using these new methods, we cangreatly reduce false positives on suspected decryption functions.The work on reverse engineering of encrypted protocols has not only pointed out thelocation of decrypted message, but also got the length of this message. Base on these results,we pioneered the method of fuzz-testing of encrypted protocols by usingin-memory-fuzzing. We can mutate the data in the decrypted buffer base on the reversedprotocol format.We have implemented a system for locating decryption functions based on existingdynamic taint tracing platform, we also implemented a system for fuzz-testing encryptedprotocols. We have tested different kinds of software including chat software and encryptedproxy server. Compared with previous work, for most of the tested software our improvedmethod is better than previous method. In order to prove the effectiveness of fuzz-testing onencrypted protocols, we have used the method to test a vulnerability on shttpd web server.Results show that our method for fuzz-testing of encrypted protocols is correct andeffective.