Research On Cluster Fault-Tolerant Techniques Of Distributed Stream Computing For Electrified Railway Monitoring Big Data

With the continuous increase in power system capacity and scale,information flow and energy flow interact closely,forming the basis for intelligent dispatch monitoring.As a special industrial power supply and distribution system,railway power supply system has a wide range of fluctuations in operating parameters.The centralized distribution of measurement points,the obvious quantification of monitoring information seas,and the high accuracy of monitoring data require higher stability and fault tolerance of dispatch monitoring operations.In order to enhance the interaction and coordination ability of dispatch monitoring systems,a variety of new smart monitoring devices are put into use.As a result,the number of monitoring sites has increased dramatically,the data volume has grown geometrically and the structure has become more diverse and complex.Typically 100 phase-measurement devices such as a regional synchrophasor monitoring system collect 6.2 billion data points a day,and the data volume is about 60 GB.If the data is calculated by 1000 monitoring devices,the data points collected each day It will reach 41.5 billion and the data volume will reach 402 GB,which poses a severe challenge to the data throughput and operational stability of the dispatch monitoring system.In order to meet the requirements of large-scale data processing in various applications,railway power supply monitoring big data technologies such as the batch computing platform of Hadoop and stream computing of Storm framework have emerged in monitoring information processing,but the railway power supply system has power supply.Frequent load changes,large fluctuations in power monitoring data,etc.,if the fault tolerance of the dispatch monitoring system is insufficient,when the monitoring data processing delay or loss occurs,it may cause the monitoring alarm information to be delayed,missed,or even misreported.At the same time,it led to decision failures in critical faults and directly threatened the operational safety of the railway power grid.Therefore,it is urgent to carry out research on fault-tolerant processing technologies for large-data related power supply monitoring of railways.Combining with the current situation of large-scale monitoring of railway power supply,the academic community has introduced record-level fault-tolerance technology in the field of distributed fault tolerance.Unlike the checkpoint mechanism that requires high resource costs,the record-level fault-tolerance technology can record files through historical operations after a fault occurs.Rebuilding all the lost partitions in the fault can effectively reduce the additional resource overhead.The elastic distributed data set fault tolerance mechanism not only has the advantages of general record-level fault tolerance,but also has better adaptabilityand fault tolerance for data parallel applications.It is a scheduling monitoring system.The efficient and reliable processing of quantitative monitoring data in the sea indicates a new solution.This paper combines the requirements of monitoring big data processing in practical engineering applications,builds a micro batch processing platform based on Spark,and a stream computing framework based on Storm.It also implements fault tolerance of CLM lineage chains and distributed fault tolerance for real-time stream computing.Taking the railway power supply dispatching monitoring system as the research object,the cluster processing performance and fault-tolerance performance are studied.The experimental results show that the Spark-based CLM fault-tolerance method can not only reduce the average CPU usage of the cluster computing nodes,but also the network when dealing with sudden data node failures.IO consumption and disk occupancy rate can also reduce the computational time-consuming of iterative calculations.Storm-based stream computing clusters have better transaction processing performance and stability after tuning,and validate the distributed data lock tuning and security queue model parameters.The effectiveness of the tuning and research results have important theoretical and practical values for the fault-tolerant processing of the massive monitoring data of the dispatch monitoring system.