Research On Storage Optimization And Parallel Processing Of Power Equipment Monitoring Big Data Based On Cloud Platform

With the rapid development of the smart grid, intelligent primary equipment and conventional power equipment on-line monitoring get fast development and become a trend. Monitoring data is becoming increasingly large and power equipment on-line monitoring system is facing more and more huge technical challenges. This dissertation focuses on the storage and parallel processing methods for the power equipment monitoring big data using cloud computing platform and big data technology(e.g. Hadoop, ODPS, Spark, etc.), which includes data distribution strategy, parallel wave signal analysis and feature extraction, parallel join query, parallel clustering and fast pattern recognition for power equiment monitoring data.Storage mode and data distribution strategy are the priorities for big data processing. This dissertation analyzes the characteristics of power equipment monitoring big data, and storage method for monitoring data is designed based on HDFS file and HBase table respectively. In order to solve the poor performance problem caused by large traffic between data nodes when doing correlation analysis among multi-source monitoring data, a data correlation based multi-copy consistency hash algorithm is proposed. Based on the algorithm and MapReduce model, multi-source map join query algorithm and multi-channel data fusion feature extraction algorithm are designed and implemented. Performance improvement are verified by experimental results.In view of the advantage of ensemble empirical mode decomposition(EEMD) for signal feature extraction and denoising, a parallel EEMD algorithm based on MapReduce model is proposed to improve the computing performance for high-speed sampling data. In consideration of the inherent defects of the rectangular window, the local flatness-adaptive segmentation envelope reconstruction algorithm is proposed to compensate segmented boundary so that the envelope error can be reduced to a given threshold range. The performance of the proposed algorithm is verified by experiment and the relation data of error margin and border length is given in experimental results.In order to improve the scalability of the platform, ODPS platform is first used to store and accelerate monitoring data processing. A storage method for partial discharge signals based on ODPS is proposed. A parallel phase resolved partial discharge analysis(PRPD) method based on extended MapReduce model(MR2) is proposed, in which extraction of fundamental parameters, calculation of statistical characteristics and pattern recognition are conducted in ODPS platform. The performance of the proposed method is verified by experiments.Under the condition of bad weather, monitoring value can exceed the upper limit in many monitoring devices and leads to large-scale alarms in a short time. Massive alarms data should be processed quickly. This dissertation designs and implements a Spark-based KNN algorithm for fast pattern recognition of insulator leakage current data on E-MapReduce platform. The experimental results show that Spark-based KNN is faster than MapReduce-based one, and is more suitable for real-time processing tasks.