MODIS SST Fast Retrieval Method Based On Apache Spark

With the remote sensing technology for earth observation continues to develop and mature,and the promoting and applying of its products and algorithms,remote sensing image processing is progressing rapidly towards the direction of extensive application scenarios,diverse image products,high spectral resolution,high spatial resolution and high temporal resolution.Now,remote sensing image data has become a typical form of big data,which not only has the basic characteristics of big data,but also has special internal and external features.In order to study the processing method of remote sensing big data,this paper takes MODIS sea surface temperature retrieval algorithm as an example,discusses the improvement of algorithm,in-memory data model and workflow processing model,and explores the remote sensing big data processing method,based on an universal parallel computing framework,Apache Spark.The research work in this paper mainly starts from the following aspects: SST retrieval algorithm that is suitable for fast calculation,remote sensing big data process which is based on in-memory data model and data processing workflow that is oriented to IO optimization.First,by comparing different algorithms and taking the needs of fast calculation into account,the SST retrieval algorithm is constructed with strong parameter synchronization and integrity,less input interruption and external dependences,in the form of fitting algorithm parameters and expansion of the algorithm expressions.Second,by comparing the differences between remote sensing big data and text big data,an in-memory model for remote sensing image based on resilient distributed dataset was proposed,and the parallel computing optimization was carried out based on cluster computing technology.Third,IO optimization-oriented data processing workflow aims at improving the execution efficiency of the whole process of the algorithm.By combining different workflow patterns,discrete remote sensing image processing steps are connected,and data access efficiency is optimized between and within steps.In accordance with the above algorithm expression,data model and workflow patterns,this paper designed and implemented the MODIS SST fast retrieval method based on Apache Spark.This method greatly improves the time efficiency of MODIS SST retrieval algorithm,and effectively saves the memory and disk space at the runtime of the algorithm.Under different execution modes,the time efficiency of this method is 13.43 times higher than that of the local one.Under different data load,this method can be stable as load adding.The research results show that the workflow method based on Spark,can effectively improve the execution efficiency of the MODIS SST retrieval algorithm,by improving algorithm,constructing inmemory data model and integrating discrete processing steps,provides a reference for remote sensing big data processing.