Design And Implementation Of Scientific Research Data Importing Module

The pace of informatization is increasing rapidly. Implementing "Researchmanagement database", as a means of information construction, to improve informationmanagement status of current research and management departments, has very importantsignificance. Informatization has an important early work, the data building, whichcontains regulating data in different formats from different places, and then organized in auniform manner, and finally store in the database.Start from the demand, the data warehousing work is divided into two parts: dataacquisition and data processing. Based on the characteristics of the existing data sourcedata acquisition is divided into two modes of importing and crawling. This paper analyzesthe characteristics of importing and crawling separately then designed and implemented acode framework. For data processing, three representatives as an entry point to detail thesource of this problem, the solution is to set forth the data processing needs to payattention to all aspects of. Finally, through the detailed correctness testing andperformance testing, the reliability and stability of the system are verified.Technically, all the work in this article is based on JavaEE platform using Hibernateand Spring Framework, and transaction processing technique to ensure accuracy andconsistency of the data warehousing. For the data crawling aspects, we use HttpClient tovisit the external sites with java thread pool to execute in parallel crawling task to improveefficiency. Through the appropriate use of design patterns, the entire code framework hasa good design and scalability.Framework to achieve in the subject has an important role to play in the actual workcompleted in advance of the data construction work, guiding significance for thesubsequent construction work.