Research And Practice On Automatic Information Extraction In Vertical Search

The rapid development of the Internet brings about the explosion of the information on the WWW. Due to the enormous information, the gerneral-purpose search engine can't meet the individuation requirement to information searching for people. So, research on search engine based on certain domain, also named vertical search engine, is becoming an important topic. Corresponding to general-purpose search engine, vertical search engine supplies more professional,accurate and in-depth results. The spider of vertical search engine need to predict and then judge the web pages' relation to the topic before it retrieve them. Only the web pages relative to the topic can be reserved to server.The main difficulty of researching vertical search engine focuses on automatic information extraction. The task of information extraction is to obtain the content in which users are interested from the web pages. To solve the defect of the information extraction system of vertical search engine, I invent a method of antomatic information extraction based on vector model of slots. The model can define the strutural relation and other visual character between slots, which can help me to fill the slots accurately.The method can study from the web pages tagged by users, and then builds the "slots" which users are interested in. The method computes a consistent coefficient which represents the character of structure,visual and context between two slots. All consistent coefficients can compose a standard vector. At the same time, in the process of learning, it can build some rules from the tagged content, and using these rules, it can eliminate many useless text fregments. When needs to extract something from an unexpected web site, it will get some pages and compare to each other and continue to eliminate the useless text fregments. After that, the method compute the consistent coefficients between the fregments and get many results of vectors. I choose the vector which is most similar to the standard vector and the corresponding text fregment is filled to the slots.