Research On Anomaly Detection Method Based On Multi-resolution Grid

Anomaly detection is an important data mining method.Its goal is to discover objects that are different from the majority of objects.These objects are called outliers.In practical applications,outliers often contain a lot of important information,and it is of great significance to find outliers in advance to avoid unknown risks and improve data quality.Anomaly detection technology has been widely used in many fields such as network intrusion detection,industrial fault detection and credit card fraud detection.However,in the current big data environment,traditional anomaly detection algorithms are faced with problems such as strong sparsity of high-dimensional data,slow computation speed of massive data,and difficulty in anomaly definition under different scenarios.Aiming at the above problems,this thesis proposes an anomaly detection method based on multi-resolution grid.The main work contents and innovations are as follows:(1)Traditional anomaly detection algorithms often need to adjust different parameters for different data to achieve the corresponding detection effect.In the face of large data,the detection time efficiency of existing algorithms is also not satisfactory.In this thesis,an anomaly detection method NA based on multi-resolution grid is proposed.Firstly,a submatrix partition parameter with good robustness is introduced to divide highdimensional data into several low-dimensional subspaces,so that the anomaly detection algorithm is carried out on the subspace,so as to ensure the applicability of high-dimensional data.Then,through the multi-resolution grid partitioning from sparse to dense,the local anomaly factors of data points in different scale grids were comprehensively weighed,and the score ranking of global outliers was finally output.Experimental results show that the new submatrix partitioning parameter is robust,the method can adapt to high-dimensional data well and can obtain good detection effect on multiple public data sets.(2)The feature processing module of NA algorithm needs to introduce the sub-matrix partition parameter.Although the parameter has good robustness,it still needs to be set manually.In this thesis,we introduce the idea of graph,and propose a feature grouping method suitable for anomaly detection,which can not only automatically divide the highdimensional data into several low-dimensional subspaces,but also keep the original feature information as much as possible.The experimental results show that this feature preprocessing method not only provides a non-parametric feature processing scheme for NA algorithm based on multi-resolution meshes,but also has wide applicability and good performance.By preprocessing data with this method,the performance of several traditional anomaly detection methods has been improved to varying degrees on multiple public highdimensional data sets.