Research On The Parallel Computing Model Of Soil Erosion Based On Hadoop

Soil erosion model is an effective technical tool to guide soil and water conservation and agricultural production by evaluating the condition of soil erosion and the influence of soil erosion on the environment and the long-term productivity of agriculture.At present,a series of lumped and distributed models are developed in the field of soil erosion at home and abroad.These models are suitable for different space-time basin units,and are applied to the prediction of soil erosion and soil and water conservation planning,monitoring and evaluation in different regions,which have achieved fruitful results.With the rapid development of geographic information technology,high precision geographic information data related to soil erosion model calculation can be obtained by using three dimensional laser scanning technology,which puts forward higher requirements for the refinement of soil erosion and high precision simulation.The serial computing mode of traditional PC or workstation can not fully meet the needs of fast data access and analysis and computation of massive high precision data.It is necessary to carry out fine simulation of high precision soil erosion under the framework of parallel computing.Existing parallel computing models(such as MPI,OpenMP,CUDA,etc.)have many defects in fault tolerance,load balancing,scalability and other aspects,and need special parallel devices(such as multiple CPU and graphics cards that supports GPU)and software programming interfaces,which restrict its application in the field of soil erosion fine simulation.In view of the above problems,this paper based on the Hadoop parallel computing framework,combined with the MapReduce parallel computing model,applied the Java programming language,selected the annual sand production experience model of soil erosion,abstracted and designed its parallel computing strategy,and developed the parallel operator of soil erosion model.Parallel computing method of terrain and precipitation factors in soil erosion model were designed and implemented on this basis.The Python script was applied to integrate the parallel computing method of soil erosion model with GIS system,and the parallel computing tool of soil erosion model was constructed.Under this framework,the segmentation-mergement parallel strategy design of the soil erosion model calculation,parallel algorithm design and implementation of the model factor,development of parallel computing tools and its integrated application with GIS and parallel performance analysis were studied.The main contents include:(1)Design of the parallel calculation strategy for soil erosion model.The main factors involved in soil erosion model were analyzed.According to its calculation requirements,three kinds of operation tasks were abstracted into local operation,neighborhood operation and global operation.According to the features of each operation type,four kinds of data segmentation strategy,row segmentation,column segmentation,block segmentation and unit segmentation were designed.Combined with the result of data segmentation,a computing result data merging method was designed,and a parallel strategy algorithm was designed based on Hadoop interface.(2)Design of the parallel algorithm for soil erosion model.By analyzing the calculation method of the influence factors of soil erosion model,and in combination with its different parallel computing strategies,five parallel operators of spatial interpolation,slope,flow direction,slope length and rasterization were abstracted,and their parallel algorithms were designed and realized.On this basis,in view of the parallel calculation requirements of soil erosion model factors,by combining and invoking the corresponding parallel operators,the parallel computing functions of each factor were designed and realized,and the parallel calculation of total soil erosion were completed then.(3)The construction of the parallel computing tool for soil erosion model.From the viewpoint of parallel computing function call and its integration with GIS,combined with SSH protocol and ArcPy,a parallel computing toolbox for soil erosion model were constructed with Python.The communication and interaction based on ArcGIS platform and Hadoop cluster were realized,including functions such as upload and download of local data and Hadoop distributed file system(HDFS),parallel computing instruction generation and remote calling of soil erosion model.Taking the basin of Gushanchuan in the Loess Plateau as an example,the simulation calculation of soil erosion was carried out by using the parallel calculation tool of soil erosion model,which verified the feasibility of the parallel strategy and algorithm design of soil erosion model.The parallel computing performance and efficiency of soil erosion model were analyzed by combining the index like speed-up ratio and calculation efficiency.The parallel calculation method and algorithm of soil erosion designed in this paper can carry out the rapid analysis and calculation of massive high precision model data,and integrate with the GIS platform organically.It has the advantages of flexible and easy to use,and provides a basic support for the application of high precision data for the fine simulation of soil erosion.