Anomaly Detection Of Data Flow Based On Time Series And Clustering

With the development of computer technology and control theory,the level of modernization is getting higher and higher,which deduced large-scale data in almost all fields.These continuously large-scaled data with very fast arrival rate is called data flow,which is different from the traditional data.The data flow generated during the industrial production is the basis of all commands issued by the industrial control system.Abnormal or erroneous data will cause the system to issue erroneous commands and perform erroneous operations,which may affect the system’s internal and the interrelationship between the sub-systems.These abnormal data also may trigger a system failure,even endanger people’s lives and safety,bring serious environmental pollution,and result in huge losses.Based on the methods of the time series and clustering,this paper studies the technology of anomaly detection in the data flow.The main contents are as follows.Firstly,this paper reviewed the progress and significance of data flow and data anomaly detection.The methods of anomaly detection are also shown in this section,which provides the basis for developing anomaly detection technology in data flow.Then,both time series and clustering are presented in this paper to detect the abnormal points in data flow.For the time series anomaly detection,complete and clear detection methods and procedures are given.Especially the idea of making the confidence interval as the detection standard enhances the feasibility of the method.For the clustering anomaly detection,this paper summarizes the specific steps of the detection,and proposes a new idea of discriminating the clustering results by using the adjacent distribution density as the detection threshold.Then the two methods are applied to the simulation data flow of Tennessee-Eastman chemical process.Using the time series anomaly detection method,the paper successfully detected the abnormal points in the simulated data flow.These abnormal points includes the step fault(fault 01),random variable fault(fault 11),slow offset fault(fault 13)and valve sticking fault(fault 14).Using the clustering anomaly detection method,we detected other abnormal points,which are step fault(fault 01),random variable fault(fault 11),valve sticking fault(fault 14)and valve position constant fault(fault 21).Thus all types of faults were detected.Finally,this paper developed a data flow anomaly detection platform based on Django Web framework.The platform integrates the time series and clustering detection methods to facilitate the automatic detection of abnormal points in data flow.