The Research And Implementation Of A Network Data Redundancy Mode Analysis System

With the development and popularization of the Internet, the Internet traffic is increasing fast, the network bandwidth which Internet service required is also increasing. As basic support service for Internet, CDN network carries the majority of the world’s Internet traffic, Data Redundancy Elimination (DRE) technology has already been applied to a certain extent in the WAN acceleration and single-site redundancy elimination, but data redundancy elimination has not been applied to the data transmission in CDN network. Analyzing data transmission redundancy mode feature in CDN network to guide the practical application of DRE technology in CDN network with the aim to improve the efficiency of data transmission is an urgent problem to solve.In this paper, based on laboratory project "The research on the application of data redundancy elimination in CDN network traffic compression", the design and implementation of each module in network data redundancy mode analysis system for CDN network traffic has been completed. The system is divided into data preprocessing module, base analysis modules, redundant pattern analysis module and results analysis display module, input data from CDN network is processed into data files divided by time period in the preprocessing module, the data module analyzes and extracts of various characteristics of the network packets and decide the data field to differentiate clients, redundant mode analysis module complete the analysis for all the characteristics of the data redundancy mode, then conclude the impact to DRE compression of technical parameters and the time period, finally analysis display module handle the data generated by the system and display them.This thesis analyzes the CDN network traffic redundancy elimination characteristics, studies the existing redundancy elimination scenario. Second, the core part of the DRE caching technology dictionary including replacement algorithm, dictionary cold mode, and dynamic static mode are studied and summarized. Rabin Karp algorithm that DRE technology used for fingerprint generation part and MAXP algorithms are summarized. On this basis, the author improves the drawbacks of existing solutions, then designs and implements the network data redundancy mode analysis system based on the understanding of the DRE technology. System design thoughts, technologies that the system used, system processes and the modules of the system are introduced comprehensively then. Finally, this paper describes a detailed analysis of the conclusions drawn from the analysis and results with the use of network data redundancy mode analysis system on sample network data.Based on the CDN network data redundancy mode analysis needs, the DRE algorithms and data content type identification methods for HTTP network have been applied in the analysis of data redundancy mode in this paper, and a network data redundant mode analysis system has been completed. The system implements the flow for the redundant mode analysis for large data of the network. With a certain value in engineering, this system can be used in the research field of redundancy mode analysis for specific network data.