Research On Storage,Management And Processing Method Of Distributed Optical Fiber Sensing Monitoring Big Data

In view of the faults of power transmission cables caused by wind dance and external force pulling and damage,as well as the severe oil and gas theft in long-distance transportation pipelines of oil and natural gas,it is necessary to carry out safety monitoring on the status of power cables and gas transmission pipelines to reduce economic losses and avoid unnecessary social hazards.Distributed optical fiber sensing has great advantages in real-time security monitoring.However,all-weather,all-distributed monitoring applications will bring massive monitoring data,which brings unprecedented challenges to the storage,query,access and processing of monitoring data.Based on the application requirements of power cable safety monitoring and pipeline safety monitoring,the distributed optical fiber sensing big data reliability storage and real-time query access are realized respectively by using HDFS and HBase under Hadoop,the distributed system infrastructure.Spark technology is used to carry out efficient statistical analysis on monitoring big data,so as to judge whether there is abnormal wind dance in power cables.Spark ML machine learning component is applied to classify the pipeline disturbance state.The performance of the traditional method and the scheme proposed in this paper are compared through experiments,and the efficiency of the scheme is verified.This article mainly completes the following aspects of work:(1)The current situation and significance of applying big data technology to solve the problems existing in distributed optical fiber sensing power cable and pipeline safety monitoring are analyzed.(2)Considering the characteristics of distributed optical fiber sensing data and big data processing technology,an overall framework for storage,management and processing of distributed optical fiber sensing big data is constructed,which adopts a three-layer architecture of data acquisition layer,data storage management layer and data processing layer.Combined with practical application scenarios,Cloudera big data cluster is built.(3)In the data storage management layer,based on HDFS,a high reliability and fast storage scheme for optical fiber sensing monitoring of big data is designed,and the advantages of high throughput and fast storage are verified by writing/reading(I/O)experiments.Based on HBase,a real-time query scheme for large data monitoring by optical fiber sensing is designed,and experiments verify that the query speed is in the "second" level.(4)For the data processing of statistical analysis,in the field of distributed optical fiber sensing power cable safety monitoring,MapReduce and Spark are respectively used to preprocess and extract features of the original data collected by the optical fiber sensing system,so as to judge the presence or absence of wind dance abnormalities.Through experiments,the data processing performance of Spark,MapReduce and single machine is compared,and the Spark rate can reach more than 10 times that of MapReduce and 30 times that of single machine.(5)For the data processing of complex iterative calculations,in the field of distributed optical fiber sensing pipeline safety monitoring,a method of data preprocessing and feature extraction is designed,and a pipeline disturbance pattern recognition scheme based on Spark ML decision tree,random forest,support for various classification algorithms such as cross product and XGBoost is constructed.The prediction accuracy is obtained through experiments,and the speed of Spark and single machine in machine learning is compared.Experimental data show that Spark speed is50 times higher than that of single machine processing.