Research On Streaming Calculation Of Coal Mine Belt System Index And Shuffle Tuning Algorithm

In recent years,with the improvement of coal mine informatization,system equipment such as mining,excavation,lifting,transportation,and communication has produced massive amounts of safety production control data and operation management data during the production process.In the era of smart mines,based on These data build a suitable coal mine big data streaming computing platform,tap the internal value of the data,realize the streaming calculation and analysis of various performance indicators of safety production equipment,and provide decision-making support for the lean management of coal mines,so as to explore effective cost reduction and efficiency increase The operation mode is of great significance to the development of coal mining enterprises.The paper takes the belt system of Shuanglong Mine in Huangling,Shaanxi as the research object,and proposes a streaming computing system for the performance indicators of the main coal flow transportation equipment.The system uses the Kafka framework to complete the monitoring function of the coal belt system data,and realizes the belt system alarm based on Spark Streaming.Flow calculation of data indicators,OEE,capacity and energy consumption indicators,performance indicators and reliability indicators.Based on on-site operating data,the visualization of the results of index streaming calculations is realized.In the Spark Streaming calculation process,in view of the data skew in the Shuffle phase,which leads to the reduction of cluster calculation efficiency,a custom partitioner based on the consistent Hash algorithm is proposed.The test results show that the custom partitioner calculates 10 batches of data with uneven key distribution,and the average calculation time for data skewed batches is 22.9s less than that of HashPartitioner.The time-consuming Z score comparison proves that data skew is no longer the main factor affecting the calculation time after the cluster calls the partitioner.The custom partitioner based on the consistent hash algorithm proposed in this paper can optimize the data skew problem in spark calculations,and the computing efficiency of the cluster is increased by 16.64%when facing skewed data tasks.The paper combines Kafka,Spark Streaming,SSM and other technologies,and takes the belt system of Shuanglong Coal Mine in Huangling,Shaanxi as an example,and establishes a flow calculation system based on coal mine belt system indicators.The calculation results can be used for lean management of coal mine belt system equipment.Provide data and decision support to improve the overall efficiency of the coal mine belt system.The research results can provide certain reference value for the smart mine construction and enterprise lean management of the Shuanglong Coal Mine in Huangling,Shaanxi.