| With the diversification of remote sensing imaging methods and the continuous enhancement of data acquisition capabilities,remote sensing data has shown the characteristics of multi-source and big volume,and the spatial resolution,spectral resolution and temporal resolution of remote sensing sensors have been continuously improved,the era of remote sensing big data has arrived.Remote sensing big data makes it possible to conduct long-term,large-space and high-precision earth observation research.However,remote sensing big data also poses new challenges to traditional remote sensing data processing methods.When dealing with the processing tasks of massive remote sensing data,it often takes a lot of time and material resources,traditional data processing methods have been unable to meet the needs of the analysis and application of massive data.In general application scenarios,remote sensing image processing involves multiple steps,such as data preprocessing,map algebraic calculation and machine learning classification.These algorithms have different characteristics and calculation methods,leading to great differences in the problems faced in the era of remote sensing big data.Therefore,in the practical application of remote sensing big data,it is necessary to fully consider the algorithm characteristics under different calculation steps,and according to the efficiency problems faced by different types of algorithms,propose a comprehensive remote sensing big data calculation method to solve the problems faced in the practical application process.In this paper,desertification information extraction is taken as a typical application scenario of remote sensing big data,through analyzing the problem faced when image preprocessing,map algebra calculation and machine learning classification are performed,image,tiles and pixel three kinds of granularity distributed computing method was proposed,to improve the efficiency of remote sensing image distributed computing.In fully consideration of remote sensing image characteristics of spatial data and in-depth analysis of large data related research,to improve the big data calculation efficiency under different constraints method research.The specific research contents and conclusions are as follows:(1)In the pre-processing stage of remote sensing image,to solve the difficulty in supporting the mainstream big data computing framework for historical remote sensing algorithms,and the difficulties of parallelization,poor portability,and inability to meet the rapid processing of massive remote sensing data have been re-developed,the author puts forward using Docker container mirror wrapping and running remote sensing algorithm,image management and distribution strategy are set according to the characteristics of remote sensing algorithm.Based on Kubernetes cluster container management engine,we could encapsulate distributed deployment and operation,to achieve the whole scene for minimum computing granularity of distributed processing,improve the history of remote sensing algorithm computation efficiency of the program.In the experimental part,the atmospheric correction program is encapsulated as an example,and the experiment is carried out under the conditions of different number of computing nodes and cluster parallelism.The result showed that expanding nodes can greatly improve the computing efficiency of the computing cluster.At the same time,choosing a suitable program to run parallelism can also effectively improve the efficiency of distributed computing.(2)For map algebra,the localization rate of data is one of the main factors affecting the efficiency of distributed computing.In order to solve the problem that the spatial characteristics of remote sensing images are not fully considered in the distributed storage,which causes data involving adjacent information distributed computing with the low localization lead to low efficiency of calculation.On the basis of research and analysis of different data segmentation methods and spatial index technology,we proposed a tile data distribution method,which fully considers the spatial information,dimension and cluster resource characteristics of remote sensing data,to ensure that the space adjacent tiles are stored to nodes with similar physical locations,reduce the cost of data migration during the calculation,and improve the efficiency of involving the adjacent information distributed computing algorithm.In the experiment,the efficiency improvement after optimized data allocation was compared under different number of nodes and data volume,and the results showed that the more nodes and data volume,the higher the adjacent computing efficiency after optimized data allocation would be,which proved that optimized data distribution plays an important role in the improvement of computing efficiency in distributed computing.(3)In order to solve the problems of complex implementation of distributed remote sensing image machine learning algorithms and the difficulty of combining and applying various machine learning models,we implemented distributed computing with the minimum granularity of pixels based on Spark framework.It supports the entire distributed machine learning algorithm library and provides the ability to combine multiple machine learning models on the same platform.Through experiment,we compared the distributed computing efficiency of different nodes and different data volumes.The result shows that when the cluster resources are limited,the increase in data volume will lead to a sharp increase in computing time.By enhancing the hardware,it can effectively improve the computing efficiency,and reduce the impact of data volume on the calculation time.(4)The remote sensing big data calculation method proposed in this paper was applied to the specific practice of desertification information extraction,and the rapid extraction of desertification information supported by cloud computing and big data technology was realized,finally the desertification information extraction in the northwest region of Mongolia was finally completed. |
PDF Full Text Request