Design And Implementation Of Network Information Collection And Processing System

In recent years, large amounts of information are widely spread through traditional media, Internet and mobile equipment at all times. For national security, the significance of grasp emergency information from variety of information is self-evident. However, the existing emergency monitoring work is completed mainly through the reading and organizing to the report of several Internet media by analysts. The situation that the large amount of data and rapid update speed determine that emergency could not be discovered, obtained and organized manually in time. It can be seen that the need of this system is very urgent.The primary works in this thesis are presented as follows:(1) By taking a full consideration to the needs of flexibility, versatility and efficiency, the B/S architecture design principle as the system architecture is determined, the framework of the network information collection and processing system is planed, and logic modules of system functions, the database structure and the user interface are designed;(2) Based on Heritrix architecture, a directed and efficient crawling for the multi-source information, embed a module which can extract web page feature intelligently is achieved, the effect of the information collection platform is optimized, and the accuracy of the system processing is improved;(3) The schema of information retrival and information storage system (IRIS) distributed full-text database and the index generation algorithm are improved, the distributed mass data storage is achieved, and the needs of fast, multi-user and full-text searching are meeted.(4) User interface with good interactivity using a cross-platform development tool called Silverlight is developed, and the quality of the system application is enhanced.The network information collection and processing system in this thesis has been put into use in certain national units. This system combines the classification of emergency topic, finds and gets real-time information of emergency in the Internet, organizes emergency information effectively, establishes hierarchical alarm mechanism for the emergency, and provides search function of emergency historical cases, in order to help analysts master emergency real-time situation and make right emergency decision quickly. Since the system was put into operation, running was stable and reliable. The expected effect is achieved.