Application Research On Big Data Technology For Geological Disaster Monitoring System

In our country,geological disasters occur frequently.Due to the increasing number and damage of landslides and other disasters,it has promoted researchers to study the rapid development of geological disaster monitoring techniques.This dissertation aims at the data mining of the historical data provided by the geological hazard monitoring system and the use of the association rules to analyses methods to conduct an in-depth analysis of the historical data to achieve better prediction function.This paper is mainly to improve the traditional Apriori algorithm,so that it can be more suitable for the deep mining of massive data in geological disaster monitoring systems.In view of the shortcomings of the traditional Apriori algorithm,this paper introduces a transaction compression algorithm in the mining of association rules.Under the vertical data format,it optimizes the steps of mining frequent itemsets.And for a single computer in some aspects there is still a lack of performance and can't better deal with the drawbacks of large-scale data,proposed based on distributed programming model improved MEC-Apriori.The new algorithm(MapReduce-ECLAT-Compress-Apriori)and the data provided by the geological disaster monitoring system verify the correctness of the algorithm.The following is the main work of the thesis:1)The basic idea of Apriori algorithm and ECLAT algorithm in association rules algorithm and the algorithm steps and shortcomings are analyzed.2)In order to optimize the Apriori algorithm,the characteristics of transaction compression are introduced into the improved algorithm,which optimizes the time performanceof the improved algorithm in mining frequent itemsets.3)Based on the ECLAT algorithm,we use the hash technique and transform the horizontal data into vertical data to optimize the time efficiency of the traditional Apriori algorithm in mining frequent itemsets.4)A MEC-Apriori algorithm based on MapReduce is proposed,and the distributed computing framework is used to optimize the time performance of the traditional Apriori algorithm.Finally,experimental analysis using the retail data set in UCI database.5)In order to further verify the feasibility of the algorithm design in this paper,and set up a hadoop cluster environment.Experimental results show that the proposed parallel algorithm has significantly improved the temporal performance and mining frequent itemsets,and successfully solved the time efficiency of the traditional Apriori algorithm in mining frequent itemsets and candidate itemsets low disadvantage.And the improved association rule algorithm has obtained potential value information in mining geological disaster monitoring data.