| With the geometrically increasing of network information resources, it has become more difficult to find the required information accurately and rapidly by using search engine, generally for two reasons. First, it is difficult for the traditional search engine to cover all the network resources, and second, the search results usually are thousands upon thousands, so user cannot find the right information which is hidden in them. The emergence of the vertical search engine becomes a milestone of the history of search engine.This paper introduces the basic theory and architecture of the vertical search engine and presents a vertical search engine of technology information domain oriented. It involves web crawler technology for vertical search engine, data minig of semi-structured web based on the hyperlink, the implementation of inverted indexing and full-text search. The work can be divided into four aspects: 1. Implement a desktop metasearch program which is used to get the domain related website's URL. 2. The design and implementation of web crawler for vertical search engine. This crawler is different from the usual one. First, it is domain related and only crawls the domain related webs. And then, by analyzing the hyperlinks, the crawler catches and automatically classifies the normal structured webs which will be processed by the subsequent steps. 3. Web filtering and structured information extraction. Filter the web pages obtained by the crawler, extract the useful information according to the field, and then save the information to the local disk in the form of XML. 4. Inverted index establishment and the implementation of retrieval interface. The goal is to find efficient data structure to establish inverted index in order to shorten the time of index and query. The system supports the combination of basic query logic in order to meet user's demand for advanced retrieval and real-time feedback mechanism which allow user's faster access to the information they need.Technical information coveres a wide scope and involves all sectors, so it is strongly needed by many users. The vertical search engine system we proposed will provide users with comprehensive and accurate information which can help users constantly meet new challenges, seize the development opportunity and make a correct decision. |
