Large-scale Atmospheric Remote Sensing Inversion Calculation Platform Based On Spark

The atmospheric environment directly affects human health.The quality of the atmospheric environment in China has attracted extensive attention.The public,researchers and government departments pay close attention to the quality of the atmospheric environment from different angles and depth.Due to the limited number of sites,the traditional site-based monitoring methods had limitations on the scope of monitoring.The satellite remote sensing observation has the advantage of regional continuous coverage.It can effectively make up the deficiency of the existing foundation monitoring methods on the regional scale.And it is a necessary means of atmospheric environmental quality monitoring.The temporal resolution,spatial resolution and spectral resolution of satellite remote sensing have been gradually improved.While it has the characteristics of large data,it also brings challenges to the fast computation of satellite remote sensing retrieval.At the same time,the different inversion methods and algorithm upgrade require the inversion platform to have flexible mechanism to adapt to process expansion.In order to solve these problems,this paper studies the modeling method of large-scale remote sensing retrieval process,and combines with the Spark distributed computing platform,established a large-scale atmospheric remote sensing inversion calculation platform with userdefined process.Firstly,the definition and processing frame of the retrieval process for atmospheric remote sensing are presented.On this basis,a process meta-model for atmospheric remote sensing retrieval is designed.Secondly,based on the domain knowledge of atmospheric remote sensing,a method of checking and transforming the process model based on domain knowledge base is proposed.The method verifies the user-defined atmospheric retrieval process and transforms the domain-oriented process model into the business-oriented process model.It ensures the high availability and scalability of the whole process.Finally,a method of optimizing Spark parameters based on XGBoost regression model and genetic algorithm is proposed.This method can optimize the parameters submitted by Spark tasks and improve the performance of Spark tasks.On the basis of the above research,it has been verified by relevant experiments.The“Arsiffe Platform”,a large-scale atmospheric remote sensing inversion calculation platform based on SPARK,is designed and implemented in this paper.Then display some of the pages.The platform aims to assist users in defining processes and accelerating atmospheric remote sensing inversion.Through experiments on this platform,the results show that all the research methods of this paper are feasible.And the large-scale atmospheric remote sensing retrieval platform based on Spark in this paper has certain application value.