Research On Efficient Abnormal Detection And Flow Moment Estimation Based On Data Center Network

Network traffic measurement is an important part of network management.It plays an im-portant role in network security.Network events such as network congestion,DDOS attacks,and worms can be identified by effective measurement of network traffic.However,with the explosive growth of network data and the huge increase in network transmission rate,the tra-ditional network traffic detection method faces many problems,and accurate counting under limited storage space becomes a challenge.To this end,many experts and scholars have car-ried out in-depth research on network flow algorithms,and have obtained rich research results.There are measurement algorithms based on packet sampling,flow-based counter measurement algorithms and flow-based Sketch measurement algorithms.Among the three mainstream net-work flow algorithms,CountMin Sketch and Count Sketch are representative,and the summary counting data stream algorithm Sketch algorithm can fully utilize limited memory resources and provide high calculation accuracy.It has received much attention and is widely used in big data,network traffic measurement and other fields.However,as far as the actual situation in high-speed networks is concerned,there are still two major problems in current algorithm research.First,for the processing of network stream data,these three types of mainstream algorithms have their own advantages and disadvantages,even stream-based Sketch,faced with the problem that memory utilization is not efficient and processing speed still has room for improvement.Second,for various emergencies in the net-work,the general algorithm can only provide very limited network flow indicator information.If multiple network events need to be detected,it will need more algorithms to process.Therefore,based on some existing data flow algorithms(e.g.,Univmon),this paper first proposes a highly compressed shared counter structure CountMin virtual active counter(CM-VAC)to further adapt to the needs of limited storage space and high-speed measurement.The CMUnivmon algorithm in this paper is realized the detection of multiple indicators of network traffic,thus improving the versatility of data stream algorithm.The contributions of this paper mainly include:(1)In view of the low memory space utilization of some current network flow algorithms,the processing speed can still be further improved.This paper combines the existing algorithms such as CountMin Sketch and Virtual Active Counters(VAC)to design a key technology,Minimum virtual active counter(CM-VAC).The algorithm greatly reduces the memory space required by compressing the 32-bit counter into 8-bit memory and letting each counter be shared by multiple streams.At the same time,the probabilistic update counter and the segmentation of the hash function value in the algorithm greatly improve the insertion and query rate of the network data.(2)In order to effectively detect the anomalous traffic events in the network,most of the current network flow algorithms can only focus on one of the measures in network traffic monitor-ing.This paper improved the original Univmon algorithm structure and proposed a Com-pressed Memory Universal Monitoring(CMUnivmon)with a very compact memory space.The algorithm uses the underlying CM-VAC structure to calculate the overall traffic infor-mation in the network.The Minimum heap in the algorithm are used as hotspot filters to further improve the insertion speed of the algorithm and the detection accuracy of the Heavy hitters event.(3)Based on the theoretical analysis,this paper tests the performance of all the proposed algo-rithms by using CAIDA real data stream trace information.The superiority of the proposed algorithm is verified by experiments.At the end of this paper,based on the general flow algorithm CMUnivmon proposed in this paper,we combine with Intel DPDK development kit,MySQL database and Flask network framework design a simple data center host-based network traffic measurement prototype system.The simple traffic simulation test environ-ment illustrates the feasibility of the algorithm in a real network environment.