Design And Implementation For A Key-value Store Based Real-time Smart Grid Data Management System

With the gradual deepening of research in the field of electric power,the expansion of the volume of massive data has made the research and application of electric power data fully entered the era of big data and cloud storage.On the one hand,the advancement of measurement,communication and sensor technology has brought rich and diverse data samples to power research and applications;on the other hand,massive heterogeneous data samples have also brought a huge impact to traditional application methods.To face the challenges,key-value-store based storage engines,as a more efficient data management method,have become a supplement or even an alternative to structured data storage solutions.However,the traditional key-value storage system still has a series of pending problems when dealing with the power data applications that undertake severe and urgent real-time requirements.First of all,due to the expansion of the collection range and the increasing diversity of collection methods,the power data has the characteristics of massive and strong heterogeneity.It is difficult for traditional key-value storage systems to maintain high data accessibility when dealing with power data in updateintensive and write-intensive scenarios under the premise of ensuring the throughput and latency;secondly,power data has the characteristics of obvious inclination and skewness for distribution and strong locality for data flow.Ignoring it will cause extra overhead and performance degradation,as well as resources waste;finally,for dataintensive applications,like machine learning and deep learning applications,the traditional data access interface will become a performance bottleneck,so that be difficult to meet the demand.Therefore,in terms of system design,we propose a real-time power data-oriented cloud storage system with functions such as real-time data acquisition,real-time data network transmission,data distributed storage retention,and high-real-time data query access.Furthermore,according to the characteristics of power data,the system can be expanded based on the three-tiers architecture,called "collection-cache interfacecentral cloud" design,to effectively improve the efficiency of data access and loading.In terms of system implementation,for the central cloud,we implement a keyvalue-store based real-time power data management system.Firstly,for the massive heterogeneous data,the Log-assisted Log-structured Merge tree（L2SM）,redesigned based on the log merge tree（LSM-tree）,is combined with the storage engine multilayer topology design,to ensure throughput and latency.Secondly,according to the workload distribution,the massive data is subjected to cold and hot separation processing and aggregation processing,to control potential space amplification and performance degradation.For the high-speed cache with access-intensive user pattern,we integrate the existing in-memory key-value storage engine Redis and redesign the data cache logic to achieve the application-oriented data access interface.For the data collection,according to different power data characteristics,the collected data can be classified and processed and transmitted in real time.In terms of system evaluation,this proposal conducts several evaluations on the real-time data processing and verifications on the system function completeness.The verification and evaluations show that,compared with the traditional key-value-store based system,the L2SM based power data collection system can efficiently achieve the functions of cached data pulling,real-time data monitoring,data query,download,visualization,and etc.In addition,compared with the traditional cloud storage system,the redesigned storage engine reduces the actual write amplification effect by 26%-37%and improves the performance by 28%-64%while maintaining the same level of accessibility.