Construction And Application Of Geological Cloud Computing Platform Based On Hadoop Cluster

The diversity of data collection methods leads to the growing data size,has reached the "geological large data" 5 "V" features,data management and analysis of the complexity of the increase,making the massive geological data efficient operation and maintenance And the difficulty of data mining is increasing,there is an urgent need for new technical means to realize the intelligent service of geological data and the potential value of mining geological data.Distributed storage and cloud computing provide a new way to solve these problems.Hadoop large data processing technology has been more and more attention of researchers at home and abroad,as a mass data storage,computing,mining technology research hotspot.This paper aims to build and build a virtualized geological cloud platform to achieve the accumulation of geological data can be shared and interoperable.Depth research and exploration of HDFS distributed file system,Map Reduce parallel programming framework and Hbase storage database in Hadoop cluster.Hadoop technology is applied to geological large data analysis and research in combination with national geological and mineral potential evaluation data.The main work of this paper is as follows:(1)Through the study of cloud computing and large data,the concept and key technology are expounded and the architecture of geological cloud platform is put forward.The open source cloud computing and storage framework Hadoop,especially the distributed file system HDFS,Frames Map Reduce and Column Storage Hbase.(2)Through the analysis of the requirement of integration,sharing and query retrieval of massive geological data,the data storage and storage cluster platform of Master / Slave architecture is built by using distributed storage technology and virtualization technology.Using HDFS and Map Reduce in Hadoop system,we provide strong technical support for the design of massive geological data storage architecture,and finally realize efficient access to geological data in high-cumulated,high-load cluster environment.(3)From the cloud storage of the Hadoop cluster,the optimization of the merge of small files in HDFS is solved,and the map reduction algorithm is used to make the merge process more efficient.At the same time,through the overall consideration of the various load factors,the use of information entropy algorithm to determine the weight value,after several rounds of load balancing,improve the system to deal with high concurrent situation,optimize the file read and write,the system efficiency has been greatly improved.(4)This paper studies the HBase database on the virtual cloud platform,and designs the rowkey according to the table characteristics of the mineral potential evaluation data,which improves the efficiency of geological data storage management and query retrieval.Through the data storage and data retrieval comparison experiment with Oracle relational database,the superiority of HBase in dealing with massive geological data is verified.