Research And Implementation Of The Large Scale Cluster Anomaly Detection Technology And Data Masking Technology

With the rapid spread of the Internet,especially mobile Internet,the amount of user data grows explosively.In order to process the massive user data,the industry began to use large scale distributed cluster for data storage and data mining.In order to ensure the security of large scale cluster,there is an urgent need for an automated anomaly detection system.In addition,the security of cluster also includes the security of user data which is stored on the cluster.In 2016 alone,there has been dozens of user privacy information disclosure incidents,the data may be used by criminals,causing damage to the safety of users.So apply the data masking technology to the large scale cluster is meaningful.This thesis describes the framework of large scale cluster anomaly detection system,including the features and classification of items in anomaly detection.For the current anomaly detection method is simple,and some servers’s performance data regularly changes,the thesis used the ARIMA model based on time series prediction for anomaly detection,then verified its fitting effect,and tested the actual effect in different situations.Then,some optimization of the large scale cluster anomaly detection system is introduced,including the process of the alarm notification module and the solution suggestion module.Finally,the data masking technology is analyzed.For the reason that the opration should be transparent to other applications,the FF1 algorithm and the FF3 algorithm are selected as the test algorithms.Then the principle of these algorithms are introduced,and the performance of the two algorithms are tested.The FF3 algorithm was implemented in the form of Spark job,and the performance of the job is optimized in several dimensions.