Research And Application Of Big DataRelated Technologies In Power Electricity Marketing System

With the rapid development of electric power enterprises and the unceasing accumulation of grid data,grid data shows a trend of explosive growth.How to select the valuable data from these tremendous data has become the key for the development of smart grid construction and power enterprises.While the rapid development of big data related technologies brings new opportunities for grid data.In the current mainstream big data technology,compared with Hadoop platform,Spark platform has more excellent iterative workload performance,high-speed computing capacity of RDD datasets and powerful storage capacity of HDFS,and behaves great advantages in mining the massive electric power data.In this paper,according to the actual demands of power companies for data analysis,Z-score standardization and FCM clustering algorithm are studied and analyzed based on the advantages of the incorporation of Spark platform with association rule mining technology.Combining Z-score standardization with FCM clustering algorithm,the power data preprocessing process is designed,and the validity of data preprocessing is verified.The FFP-growth algorithm is improved by using sparse matrix and FCM clustering algorithm,which is applied to the power marketing system under Spark platform.This paper mainly focuses on the following three aspects.First,the problems encountered in the actual data processing of the power grid are studied and some solutions are proposed to solving these problems.Big data preprocessing of the integration of Z-score standardization with FCM clustering algorithm and FP-growth association rule mining algorithm are given key studies.Second,the shortcomings of traditional FP-Growth algorithm are improved,sparse matrix storage is introduced to reduce frequent item sets gotten by a database scan,and the time of database scan is saved.The improved FFP-growth algorithm flow is designed under Spark platform,and the data are classified twice to avoid the single transaction set too big.Third,Spark platform is built up,and power marketing system is designed and implemented.Data preprocessing technology and improved FP-growth association rule mining technology are applied to power marketing data analysis system.The effectiveness of the redesigned data preprocessing process and the improved parallel association rule mining algorithm in power marketing data analysis is verified through examples.