Research On Task Allocation For Distributed WEB Crawler System

With the rapid development of computer networks,mobile networks and social networks,the quantity of information is increasing every moment.Because the information in network is numerous and locates everywhere,the net crawler system that has the charge of gathering information is facing a big challenge.Fortunately,task allocation in the net crawler system can handle this problem.This paper mainly focuses on the characters of distributed net crawler system and its task allocation problem,we propose a task allocation module using in distributed net crawler system,and analyze its performance,the main work is as follows:Firstly,this paper summarized the background,development and characters of the distributed net crawler system and task allocation problem,focused the feathers on task allocation of distributed net crawler system,and proposed the model based on MAS theory.Secondly,based on the analysis of the MAS model,CNP was used for communication between the different agents,in order to improve the efficiency,some necessary parameters was introduced,then a task allocation method using the improved contract net protocol was showed,which enhanced a better performance.Finally,discussed on the theory about mechanism design and auction,gave detailed analysis of VCG Auction,then a task allocation method based on VCG Auction was proposed,and proved the efficiency,incentive compatibility of the method.In a word,the main purpose of this paper is to handle the task allocation problem of distributed net crawler system.The result of this paper has important reference value to the study and application for researchers.