Implementation And Improvement Of Crawling Technology Of Web Application Vulnerability Scanners

In the past few decades, the Web information technology has rapidly developed. Web has become an essential part of people’s life. The widely use of the Web application system make people’s life a lot easier. However, it also brings a lot of security risks. During the development of a Web application, many companies didn’t pay enough attention to the application security risks that may exist, which results in a lot of vulnerabilities when the application putting into use and threats the safety of the Internet.This paper first describes the Web application security situation, and introduces several common Web vulnerabilities. Then it compares several vulnerability scanners from three parts, namely input vector coverage, crawling ability and authentication, and the most important factor a scanner should improve is the crawling speed and authentication ability. To improve the crawling speed of a scanner, the paper proposes a distributed crawling technology. We distribute CrawlerNodes on several hosts separately, which improves the efficiency and expandability of the scanner. The paper also analyze some key technologies in the distributed crawling module, including URL distributing, URL de-weight, DNS resolving, information collection based on authentication. The solutions of these key technologies improve the whole crawling module’s performance greatly. At last, the paper test the scanner’s scanning speed and authentication. The experiment proves that the scanner based on distributed crawlers can not only improve the scanning speed, but also collect authenticated information.