Study On Distributed Data Warehouse Of Oil And Gas Drilling Information Based On Big Data

The intelligent drilling is the inevitable development trend of the oil and gas drilling in the future.The data volume of oil and gas drilling informations are growing at an unprecedented speed,and the information storage of oil and gas information is widely distributed.How to store and utilize the massive data is an urgent problem in the oil and gas drilling operation.For example,massive heterogeneous data storage and extraction transformation mode,data synchronization method of distributed storage,method to processe the massive data and so on.In this paper,a distributed data warehouse model of oil and gas information based on Hadoop is proposed,which provides a guide for solving the above problems.The work of the paper mainlu include four aspects as follows:First,the distributed data warehouse model under big data environment is proposed.Based on the characteristics of oil and gas information,combined with the knowledge of Hadoop theory,the storage of massive oil and gas drilling information is realized through HDFS.Hive data warehouse realizes large-scale parallel query of data.MapReduce realizes large-scale parallel operation of data.HBase provides real-time information query service.Sqoop technology is used to interact with relational database.Second,the design of distributed data warehouse of oil and gas information based on Hadoop.It includes the design of data warehouse theme and topic domain,the design of fact table and dimension table,the design of data model and the design of data granularity,and discussion of the process of data extraction,conversion and loading.Then,according to the characteristics of mass oil and gas drilling information,each part is optimized,including optimization of HDFS storage,optimization of MapReduce operation,optimization of data query of distributed data warehouse of oil and gas drilling information based on Hive and optimization of HBase storage.Finally,design and realization of the distributed data warehouse system of oil and gas drilling information based on Hadoop,and then a Hadoop cluster is built and distributed storage and analysis of drilling data is carried out.And the traditional data warehouse is set up as the contrast group,and the drilling data of different data is queried and analysized respectively.A polygonal graph with variable data size and operation time is drawed to get the conclusion.