Research And Application Of Storm On Power Big Data Platform

Power system monitors every process during the power production, management and operation processes and accumulates amount of real-time data, so real-time process of streaming data has been a hot area of research. Distributed technologies of stream data process make it possible to analysis massive streaming data.Storm is an open source computing system which is distributed, real time and has high fault-tolerance performance.It is most often used to real-time analysis, online machine learning, continuous computing scenes, with good results. The topic of this paper is "Research and application of Storm on power big data platform",we aim at combining the Storm with the power big data platform.We presents treatment schemes for amount of real-time data in power system. The main contents include the following aspects:First,research the distributed stream processing technology platform and their representatives. Analyses and compares the Storm,S4, SparkStreaming in their architecture, data model and fault recovery;Study related technologies of stream data processing.like ZooKeepe、Kafka.Second, do deep analysis on storm architecture and process model; Study Storm stream processing policy, including grouping strategy and real-time data stream processing lossless algorithms. Give a topology example and verify the performance of Storm clusters in different topology and different parameters configuration,include CPU utilization, system throughput and processing latency. Determining the cause of accident in order to provide support for Storm performance optimization and system expansion.Third, integrate the Storm with power system big data platform. Knowing demand of streaming data processing in power system, we propose the framework of power big data platform which is layered and have different calculation modes; Design a streaming data processing procedure based on Storm,add Kafka for data acquisition. After data preprocessing,data is transmitted to data processing modules, the result is placed in HBase.Fourth, Storm application in power equipment condition monitoring system. Research the electrical equipment condition monitoring system, set insulator monitoring as example, analysis the streaming data processing procedure from three basic steps:data collection, feature extraction and condition assessment.Finally, achieve the entire process based on Storm, and gives the system displays.