Research And Design Of Mine Big Data Analysis Platform Based On Hadoop

With the incessant expansion of societal informatization and the country's significant emphasis on production security,coal mining enterprises have built many digital systems to ensure safe mining of coal mines.However,these systems did not have a unified plan in the early stage.They are all self-contained,and information cannot be shared between the systems,forming many "Information Islands".With the appliance and growth of modern information technologies such as fifth-generation communication technology,big data,and artificial intelligence,human society is transforming to intelligence,and coal mining is also growing in the orientation of intelligence.The construction of smart mines is an important sign.During the operation of the coal mine digital system,a tremendous quantum of data is generated.Traditional data technology is difficult to process data of this magnitude,which limits the further analysis and utilization of big data in the mine,resulting in the loss of data value,and cannot meet the safe development of coal mining.The construction of smart mines is inseparable from the analysis of massive multi-source heterogeneous data.Therefore,building a mining big data analysis platform is the prerequisite and basis for building smart mines.This article starts from the mine big data handling,and mainly advances in-depth research on the mine big data preprocessing and mine big data analysis.First,the growth and research status of smart mine and mine big data platforms and the theory of big data processing are described.Secondly,three commonly used data preprocessing interpolation methods are introduced for the processing of missing values caused by sensor abnormalities.Based on the data fusion theory,a data preprocessing method is proposed and compared with experiments.Third,the classic association rule analysis and mining method Apriori is introduced in detail and its shortcomings are analyzed.Aiming at the defect that the Apriori algorithm cannot efficiently mine association rules,an ameliorated distributed parallel algorithm is proposed and verified by experimental analysis.The improved algorithm is used to realize the association analysis of mining big data.Finally,the hierarchical structure of the mining big data analysis platform was designed and the architecture was designed.The Hadoop distributed storage and computing framework and its ecological technology are used to construct the mining big data analysis platform and realize the visual display of part of the analysis data of the coal mine.The Hadoop-based mining big data analysis platform integrates the various digital systems of the coal mine to form a whole,and uses distributed clusters to realize the unified processing and storage of multi-source heterogeneous data in the mine.It breaks the "information island" effect brought about by the independent deployment of multiple systems,and realizes the sharing of information between systems and the comprehensive analysis of data.It can provide decision-making reference for coal mine managers while reducing the input cost of coal mine enterprises,which is of great significance for realizing coal mine safety production and improving the level of intelligent coal mining.