Research On Real-time Streaming Processing Method Of Smart Grid Big Data

In recent years, with the wide use of sensor measurement technology, communication technology and computer technology in smart grid, Phasor Measurement Unit, intelligent meter collection, data acquisition and monitoring system and so on,have produced with exponential growth of data,showing the characteristics of big data on volume,variety and others, and how to deal with these data quickly is a great challenge to the smart grid. The application of big data technology in smart grid is mainly concentrated on the field of mass data acquisition, storage, analysis, visualization, and other fields. At present, large data processing technology can be divided into two modes of batch processing and stream processing. Batch processing system has the characteristics of calculation after storage, data accuracy and comprehensive requirements. Stream processing systems are often not required to be absolutely accurate, and pay attention to the dynamic production of data for real-time calculation and timely feedback results. Due to the various restrictions, such as the special nature of data stream processing and the timeliness of big data processing, the traditional real time processing technology has not been able to meet the demand, therefore, the stream processing of big data has become a hot spot in the research and industry.This paper first analyzes the source of big data of the smart grid, then focus on the features of large volume, wide variety and fast speed, it is pointed out that the monitoring data of the state monitoring device and the electric energy measurement of the smart grid is gradually formed scale data stream. Data stream has the characteristics of real time, ease of loss, disorder and infinity, the value of the data stream will decrease with time. Combined with big data processing technology, a new intelligent power grid real time data stream processing framework is presented, which is based on stream computing system, by collecting data of monitoring data source changes of system nodes, and collecting data in real time, using message subscription model to buffer and calculate data to meet the need of rapid analysis in business applications such as abnormal detection and abnormal use of power analysis.Real-time processing of data stream is a continuous process, which is essentially a continuous batch processing technology, can be set up as an hour, minute or second in the process of batch computing. Taking the anomaly detection of condition monitoring data flow as an example, the sliding window processing topology implements in the storm stream computing framework, and judges the threshold value of the time series data to improve the real-time performance of data processing, all of this provide a thinking for big data processing technology apply to smart grid. The experimental results show that, under the condition of a certain cluster size, appropriate changes the number of working processes and the number of concurrent execution threads can reduce the processing delay and improve the real-time processing efficiency in anomaly monitoring of condition monitoring.