Research On Big Data Attribute Reduction Of Power System

Along with the enhancement of industrialization and social informatization, the degree of automation in various fields is becoming more and more high, large and super large scale data came into being, which showing the characteristics of big data. These massive amounts of data in the promotion of industry development momentum, at the same time, it brings a huge challenge- data availability. In order to identify useless information from vast amounts of data, and dig out valuable information for the development of related fields, it is necessary to carry out data analysis. Data preprocessing technology can greatly reduce the data processing capacity, improve the efficiency of data analysis and processing, and attribute reduction is a very important part in data preprocessing. And because the link of operation and monitoring in power system generated by the data types of various, size of large, which is the feature of big data, so it is necessary to do reduction of attributes for using this kind of data more effectively. The existing approaches based on modern heuristic optimization techniques such as genetic algorithm have the disadvantages of narrow coverage and high complexity, and can not be applied to the field of electric power for data processing.This thesis analyzes the research status of non parallel attribute reduction algorithm and the parallel attribute reduction algorithm at home and abroad. Firstly, we analyze the performance of the existing parallel algorithm, the algorithm has defects of high complexity and easy to be lost, and the traditional non parallel attribute reduction algorithm has the memory bottleneck problem, cannot reduce the massive data. Then combined with the characteristics of big data of power system, focus on the independence of the concurrent events, comprehensive utilization the parallel advantages of Map Reduce algorithm and partial algorithm, gives the formal description of partial order reduction. Finally, gives the algorithm of partial order reduction design in power system, and according to the actual situation of different values of the power data decision attributes, two kinds of reduction schemes are given. scheme 1: if the number of decision attribute values are many, then you can select the algorithm 1 and 2; scheme 2: if the number of decision attribute values are less, then you can choose the algorithm 1 and 3; however, in order to improve the efficiency of data preprocessing, in the case of many decision attribute values, also can choose the algorithm 1 and algorithm 3.In this thesis, taking the monitoring data of a photovoltaic power generation system, transformer fault diagnosis data and the data of real time and reliability prediction in intelligent substation communication system as an example, the attribute reduction is simulated, and to test on the Hadoop platform, which shows that this reduction method of power big data is excellent. Verify the partial order reduction algorithm that proposed in this thesis can not only solve the problem of decision table information loss caused by the heuristic attribute reduction algorithm, also skip heuristic attribute reduction algorithm to compute the core process, which has the superiority on performance.