Research On Air Quality Prediction Method Based On Hadoop

Urban air pollution is currently the most serious environmental pollution problem in China.It not only causes serious damage to our country's economy,but also brings great hidden dangers to the physical and mental health of urban residents in China.Therefore,prevention and treatment of air pollution are imminent.To prevent air pollution,air quality must be forecasted and forecasted.Timely and accurate air quality prediction can not only help city managers make scientific and effective prevention and control measures,but also provide healthier and safer strategies for urban residents to travel.At present,China's air monitoring system is gradually improving,its scale continues to expand,and massive amounts of air quality data continue to accumulate.With the rapid expansion of data size,the traditional methods of air quality prediction technology have been unable to deal with these massive data.In view of the above reasons,the dissertation has developed a method for predicting air quality based on big data processing technology.The main research work is as follows:First of all,in order to meet the challenges of mass air monitoring and meteorological data analysis,in order to achieve air quality prediction,the Hadoop big data platform was built.The method is to use VMware software to create 3 Linux virtual machines on the PC and install and configure the JDK on the virtual machine,then install Hadoop software on each virtual machine and Configure the core-site.xml,hdfs-site.xml,mapred-site.xml and yarn-site.xml files,then format the name node and start the Hadoop cluster.Secondly,in order to obtain air monitoring and meteorological historical data,this article uses python to implement the web crawler technology,And from the Tianqihoubao net and TuTIempo net,it Crawled data of the major cities in China for four years,including six kinds air pollutant concentration data such as PM2.5,PM10,CO,NO₂,O₃,SO₂,and five kinds meteorology data such as temperature,humidity,wind speed,wind direction,precipitation.In the face of the cluttered data captured,data cleaning has become an important task.Only the cleaned data can facilitate air quality prediction.The data cleaning in this paper is mainly to fill in and correct some missing data and outliers in the data.In addition,non-numerical data such as cities and wind direction were numerically processed.In order to facilitate training of neural network models,data normalization processing was performed.Finally,the paper proposes an air quality prediction method based on BP neural network under big data platform.The idea is to use the MapReduce parallel computing framework in the Hadoop platform to achieve the parallel processing of the traditional BP neural network to improve the processing capacity and training speed of the BP algorithm for massive data.The paper firstly established a neural network model for air quality prediction.The model selected the meteorological factors of the day,the concentration of the six pollutants of the previous day,and the corresponding city as the input of the model.The concentration of six pollutants on the day was used as the output.The parameters of the model were determined through experiments.According to the established model,the paper designed and implemented an air quality prediction algorithm based on MapReduce-BP neural network.In summary,this paper mainly focuses on the Hadoop platform and uses the traditional BP neural network algorithm as the core to achieve air quality prediction.This method solves the problem of air quality prediction in large-scale data environments.