| With the rapid development of the Internet, data is dramatically increasing on the Internet beyond the human imagination, people’s demands for data and information is also sustainable growth. Search engine can help people to retrieve the needed information and data from the vast amounts of data, so it has become essential tool in people’s daily life, and it also affects the memory of people’s daily habits. By analyzing global mainstream search engines in current day, including Google, Baidu, Yahoo!, Bing, Sogou and so on, we found that these mainstream search engine contains three main parts:Web Crawler,Index and Front-End Search. The three parts is recognized by the industry.However, with the increasing information demand of people, the general search engine can not meet the special demands of people, because general search engine results produce large amount of data, and deep is not enough. Basically, vertical search engine vigorously developed. Due to the vertical search engine is professional, and as well as the exclusive domain of the depth, the majority of Internet users very like itPowerful search engine can not do without the support of the data, and data sources of search engine mainly through web crawler to get, so web crawler is critical for search engines. This article is vertical search engine oriented, build Chang You system, provide three services, including general search, book search, and video search. Firstly, according to the campus network situation to design the architecture of the Chang You system, polymerizing the three services at the same entrance, and making it access seamlessly by different network segment. Secondly, researching current mainstream open source web crawler, according to the demand of ChangYou to select Heritrix as web crawler prototype, and analyzing the source code, basically, highly customizing Heritrix and solving the problem of Ajax. Then, according to the Heritrix running status and characteristics, designing and implementing which is suitable for vertical search engine incremental crawl-ChangyouSpider, which is lightweight, efficient, to compensate the lack of Heritrix, thus, this paper make use of Heritrix and ChangyouSpider combination as web crawler of vertical search engine. Finally, by crawled data to verify the function of the crawler and performance in the aspect of the totality, impurity rate, aiax test, page validity. |
