| With the rapid development of internet, information publishing and dissemination have become more and more quickly. The amount of information on the network has accumulated to a very large scale, making it increasingly difficult to retrieve information. Fortunately search engines can provide users with rapid information retrieve, which is more and more used in people's daily life. Web crawler is crawl module of a search engine, whose main task is collecting internet webs. The performance of crawler system and the quality of collected webs seriously affect the service quality of search engines, so crawler system is a very important part of a search engine and it is worthy to be researched and improved. In recent years, subject to the size of the network, the common crawler undertakes more and more onerous burden. However, focused crawler which pays more attention on special topic areas has higher operating efficiency. It has been widely concerned and deeply researched in recent years. Focused crawler opens up a new path for crawler field, with high research value and practical value. This paper mainly discusses various technologies involved in designing and implementing a focused crawler system. The main research and work as follows.(1)Proposed and implemented an improved search strategy, mainly through the dynamic adjustment of threshold, so that the search process would be more flexible. This method can effectively resolve the tunnel phenomenon that commonly exists in focused crawler field. And also it can balance coverage and accuracy of focused crawler system, so that both of them will reach a higher level.(2) Proposed and implemented an improved model of the correlation calculation, mainly through compressing the vector space dimension, which can reduce the computation time complexity and improve the calculation speed without losing much accuracy.(3) Studied when the data size is too large, how the crawler re-works the news URL and how to story webs. The former is mainly through the establishment into a B-tree index structure, which effectively reduced the space required for maintaining URL queues. The latter is mainly to give up the webs stored in the database, in stead of, webs would be stored in documents by the certain format. This is conducive to the maintenance and security of data.(4) Studied various technical issues in focused crawler area, including web access, web content analysis, Chinese word segmentation and search strategies. Summing up all aspects of the above theories, designed and implemented a focused crawler system. Using this system, the improved search strategy and correlation calculation works well verified by relevant experiments data. |
