Research On Semantic Description And Storage Method Of Coal Mine Equipment Big Data Based On Hadoop

With the wide application of big data and Internet of Things technology in industrial field,coal mine production is developing in the direction of informatization and intelligence.Collecting,storing,processing and analyzing data of coal mine equipment can reflect the running state of equipment timely and accurately,which is an important basis for realizing health state assessment,life prediction and fault diagnosis of coal mine equipment.Coal mine electromechanical equipment is becoming larger and more complicated,device data is growing explosively.Moreover,the multi-source heterogeneity of data is becoming more and more obvious,which make the data can’t be shared and used efficiently.The traditional data management platform of coal mine equipment has been unable to store and process massive data,nor can it meet the needs of building smart coal mines.Therefore,focusing on the massive multi-source heterogeneous data of coal mine equipment,researching the unified description of equipment metadata in depth,optimizing the big data of equipment storage method,building the equipment data management platform are of great importance in building smart coal mine.Firstly,the characteristics of multi-source heterogeneous,high timeliness,high dynamic response,high transmission and massive about the coal mine equipment data are analyzed.Combined with the requirements of coal mine equipment data management platform,the overall architecture of coal mine equipment data management platform based on Hadoop is proposed.It is designed from four modules:data acquisition layer,data description layer,data storage layer and data processing layer.Finally,the visual operation of the data platform and interaction with users are realized.In order to realize efficient sharing and utilization of coal mine equipment data.Firstly,the coal mine equipment data set is divided,and the equipment metadata is described by RDF resource description framework.Combined with the equipment data resource instance,verify the generality and feasibility of the description method.At the same time,build the metadata ontology model and complete the ontology instance mapping,thus to improve the interoperability between coal mine equipment data.In the face of traditional storage methods can’t satisfy the storage requirements of RDF metadata about the massive device.A distributed storage model of RDF metadata which based on HBase is proposed.Using the column storage,scalability and other characteristics of HBase to complete RDF metadata storage and loading.And two storage structures are optimized to realize fast query of RDF data.Completing the reading and writing of device file data through HDFS.However,in the process of storing and dealing a large number of small file data will consume too many cluster resources and bring some problems such as low access efficiency.Through in-depth research on the storage and processing methods of small file data.A small file merge storage optimization method is proposed to improve the efficiency of small file access.Meanwhile,an optimization method of small file merging processing is designed to improve the efficiency of data processing.A secondary index is constructed for small file mapping data and coal mine equipment metadata,which realize non primary key query and improve the data query capability of the cluster.Finally,the Hadoop distributed storage system is used as the framework to complete the construction of the coal mine equipment data management platform.Make the performance test experiment about the small file data storage processing and the secondary index query on the cluster.Verify the feasibility and efficiency of optimization methods about data storage and query,which through the research and analysis of the experimental results.