Research On Remote Sensing Image Processing Based On Spark Platform

With the continuous development of remote sensing image processing technology,remote sensing image processing methods have evolved from traditional manual visual interpretation,stand-alone processing to algorithm intelligence and distributed cluster processing.Based on the Spark,a distributed memory computing framework,this dissertation deeply studies the parallel processing of remote sensing images in a distributed environment,designs a distributed remote sensing image storage management method,proposes a parallel remote sensing image scene classification algorithm,and finally designs and constructs remote sensing image processing system based on Spark platform.The main research contents are as follows:(1)Remote sensing image storage management based on distributed storage system.In this dissertation,the distributed file system HDFS is used for primary storage of the original remote sensing image data to ensure the safety and reliability of the data and avoid the destructive modification of the original data.On this basis,the columnar-oriented non-relational database is used for secondary storage of remote sensing image data,and a column-oriented remote sensing image data partitioning strategy is designed.The large-scale remote sensing image data is divided and stored in HBase,thus the distributed storage and management function of remote sensing image data is completed.(2)Research on parallelization of remote sensing image scene classification based on distributed environment.In this dissertation,by studying the problem of remote sensing image scene classification in a distributed environment,a parallel algorithm for remote sensing image scene classification based on artificial information features SURF and deep learning semantic features is proposed.First,for the remote sensing image data,the SURF feature is extracted and re-encoded using the VLAD algorithm to form standardized feature information;then the VGG16 pre-trained network is used for transfer learning to extract high-level semantic feature information of the remote sensing image.After normalization processing and PCA dimensionality reduction,the final classification features of remote sensing image scenes are fused,and then the random forest algorithm is used to train the classifier,which has achieved better experimental results.Based on the algorithm,this dissertation uses the Spark parallel computing processing platform,by designing data parallelization during remote sensing image feature extraction and model parallelization during random forest training,realizes the parallel research of remote sensing image scene classification algorithm under the Spark platform.(3)Construct remote sensing image processing system based on Spark distributed computing framework.This dissertation analyzes the remote sensing image processing process and builds a remote sensing image processing system based on the Spark platform.Firstly,HDFS and HBase are used to store the remote sensing image data,and then the GDAL remote sensing image processing library is used to complete the preprocessing tasks of the remote sensing image data.The Spark computing framework is used to provide remote sensing image processing functions by integrating other algorithm resource libraries.On this basis,through the exploration and optimization of the system parameters,a set of better system parameters was selected.Through the exploration and optimization of the system architecture,Kubernetes was introduced for system monitoring and management,and a remote sensing image processing system based on the Spark platform was constructed.