Outlier Detection And Application In Ecological Monitoring On Data Stream

Outlier detection is a very important research branch in the data mining domain which can detect the data whose behavior pattern is significantly different from other data objects.With the emerging of Internet of things technology,high-speed,infinite and dynamic high-dimensional data stream is generated in the ecological field,and the anomaly detection of ecological monitoring data is also crucial.Based on the above,this thesis conducts the following research work:1)The research status of outlier detection in data stream is investigated in detail,and the relevant methods are summarized and classified.Furthermore,this thesis analyzes and compares the advantages and disadvantages of typical data stream processing models and the corresponding anomaly detection methods,and proposes the improvement idea for the key problems faced by data stream anomaly detection.2)Due to the well-known "dimension disaster" problem,the existing detection methods for high-dimensional data stream have poor detection effect,while the anglebased method is more stable than distance-based method in high-dimensional space,however,the computational complexity is very high.Therefore,it is not suitable for the detection of large amount of data.To solve this problem,a fast anomaly detection algorithm based on vector dot product density is proposed in this thesis.The algorithm only performs incremental calculation on the affected data points in the sliding window,and utilizes two optimization strategies and a pruning rule to effectively reduce the number of distance calculation in the detection process,thus reducing the cost of the algorithm in time and space and improving the detection efficiency.The method proposed in this thesis is efficient through theoretical proof and multiple experimental analysis of simulation data set and real data set.3)An application system of ecological stream data audit based on anomaly detection is designed.Four functions which are abnormal data detection module,data audit module,data analysis module and auxiliary management module are realized in the system.The system has a practical anomaly detection method.Meanwhile,the fast anomaly detection algorithm based on vector dot product density is applied to the system.These methods not only have high accuracy of anomaly identification,but also ensure high efficiency of operation.The system is a highly universal and extensible system,which can meet the real-time anomaly detection and audit of all kinds of ecological data.Furthermore,cluster analysis technology is employed to partition the adjacent relationship of ecological monitoring stations,which makes the abnormal detection results of ecological monitoring data more accurate.In summary,compared with some traditional outlier detection methods,the method proposed in this thesis is very suitable for anomaly detection of ecological monitoring data flow in the ecological field with increasing data volume and dimension which has a certain application value in the abnormal detection of ecological monitoring data.