Application Of Hadoop Technology In Water Quality Data Analysis And Management Of Three Gorges Reservoir Area

The Three Gorges Reservoir Area is located in the Yangtze River Basin of my country.It is rich in water resources and plays a very important role in the strategy of freshwater reserves.However,the rising water level in the reservoir area gradually reduced the purification capacity of the water.At the same time,due to the increased pollution of water,the local water pollution problem is becoming increasingly prominent.With the development of the application of information technology,the amount of data generated in the field of hydrology has expanded rapidly.The frequent occurrence of sudden water pollution has made the timeliness requirements of application systems in the field of hydrology more stringent,modern and efficient.The demand for integrated construction of hydrological systems has become more urgent.Commonly used water quality analysis schemes will not be able to calculate and query massive data in a timely manner,and it is difficult to effectively obtain the water environment status.Hadoop distributed technology has the advantage of strong parallel computing ability,which can meet the analysis and storage in the case of large amount of data.Based on the distributed technology,this paper designed the water quality analysis and management system of the Three Gorges Reservoir area.The main work of this paper is as follows:(1)Combined with the advantages of wireless sensor network,such as strong real-time performance,low cost and wide distribution,the paper established the perception module of water quality analysis and management system.Then used Flume and Kafka build data acquisition and integration module,so the data was collected through Flume,and then transferred to Kafka cluster for integration and module decoupling.(2)In order to effectively evaluate and classify water quality data,the paper constructed an online analysis module.In this module,the paper improved classification accuracy by integrating classification models with stacking algorithm,and combined this algorithm with Spark Streaming framework to realize the flow processing of real-time water quality data.(3)then the paper build a data storage module based on redis and HBase cluster,ensuring the real-time water quality monitoring through redis memory database,and stored the node water quality data through HBase,which can improve the data query ability by using Rowkey design strategy,and backup the data by Hadoop cluster to ensure the data persistent storage.Second,used spring MVC architecture build the system software platform,established the back-end framework of the software platform to returns the data required by the users to the front-end page,provided the user with the regional water quality monitoring function,node data query function,node visual management function and SMS water quality level reminder function.The software platform used Baidu Map API and heatmap.js to improve the front-end page’s straightness.So the functional requirements of the system software platform can be realized.(4)After completing the module deployment of the system,the paper verified the accuracy and calculation efficiency of the system classification algorithm,and tested the function and system performance of each module.The results showed that the stacking classification model based on Spark Streaming can effectively improve the accuracy of water quality classification and ensure good parallel computing ability;and the various modules of the system could work together,the software platform could operate normally,maintaining good availability and stability.So the system can achieved the needs of online analysis and management of massive water quality data.