Research On The Understanding And Cooperation Of The Topic Crawlers Based On Multi-Agent System

The Multi-Agent system (MAS) is a kind of technology to achieve certain system targets or to solve some problems in large scale using the characteristics of Agent—autonomy, flexibility, interaction and other features. In MAS, communication is the base of cooperation and negotiation among agents. However, different Agent which stands in different platform or holds different knowledge and background always has different message format or different understandings on the concepts and structures in the message content. These directly lead to the confusion in Agent communication. Therefore, the Agents need to share knowledge and unify the message format. Agent communication based on Ontology and Formal Concept Analysis (FCA) is a method pointed out regarding the problems above. It can strengthen Agent's semantic understanding on message content and increase the efficiency of communication and cooperation, accordingly enhance the reusability, inter-operation and reliability of the software.With the rapid development of the Internet, general purpose web crawler becomes more and more powerless on extracting the web information efficiently and effectively from the mountains of the network. However, the Topic-specific crawler solve the scalability limits effectively to gather only particular pages related to specific topic, which can guide the crawl URL links and extract the landing pages with highly relevant content. This paper presents a Topic-specific crawler model based on MAS and tries to calculate the understanding between Agent Crawlers relies on Ontology and FCA. By calculating the similarity between concept and concept, concept and Agent, Agent and Agent to measure the degree of understanding, its application in the theme based on MAS to guide the crawling reptiles.This paper mainly summarized as follows:1. This paper presents a new method to measure the understanding among Web Crawlers Based on MAS. When calculate the similarity between the concepts of the Intelligence Agent Crawlers, not just natural language comparison between words, but also consider the use of ontology and its semantic relationships. That is extending the extension of the concept to the various links relations between pages and adding the semantic of ontology, such as synonymous, equal, Is-A and Part-Of, etc. to the connotation of the concept. Besides, it considers the level of the concept lattice to the similarity. To some extents, it can eliminate the ambiguity of natural language and improve the accuracy of understanding.2. A topic-specific crawler model based on MAS was designed and implemented on the JADE platform based on it, using the modified contract net protocol simulation of multi-Agent coordination. Finally, using the web test sets to evaluate the model, experimental results show that based on MAS the theme of the topic-specific crawler retrieval more relevance web pages and the relative rates have improved to some extent.