Remote Sensing Image Retrieval Based On Deep Learning Features

Remote sensing Earth observation has currently reached an unprecedented level,and the availability of remote sensing data has grown exponentially,however,we are overwhelmed by the massive data with too much unuseful information due to the limitation of data processing techniques.Therefore,how to carry out the rapid localization and intelligent retrieval of objects or regions of interest from remote sensing images has been a significant necessity of remote sensing big data processing and analysis as well as a big challenge to be solved in the field of remote sensing image processing.Content-based image retrieval performs query and search of remote sensing data via the extracted low-level visual features which is an effective method for remote sensing data organization and management.However,content-based image retrieval are facing two big challenges with respect to massive remote sensing data,which are(1)remote sensing image retrieval based on single or combined low-level visual features are unable to achieve satisfactory performance due to the characteristics of large volume,scale-dependent,multiple object classes and scene complexity of remote sensing images,and(2)it is impractical to design a feature descriptor that is appropriate for all kinds of sensor type images.Recently,deep learning has drawn extensive attraction in the literature,and has grown to be a research focus in image interpretation and recognition field.Deep learning is available for multi-level representation of image content by constructing multi-layer network structure.Considering the fact that conventional content-based remote sensing image retrieval relies on handcrafted features which are not powerful enough and thus result in unsatisfactory performance,in this article,we therefore conduct scene analysis of complex remote sensing image based on deep learning,and perform accurate and fast retrieval of massive remote sensing images via automatic feature learning.The research in this article mainly includes the following three aspects:(1)Unsupervised feature learning and retrieval method for remote sensing image.Conventional content-based remote sensing image retrieval relied on low-level visual features such as spectral(color)features,texture features,and shape features etc.,which belong to handcrafted features.However,it is time-consuming and laboursome to design a robust and effective feature descriptor.Further,the designed handcrafted features may not be appropriate for images acquired by various satellite sensors.To solve this problem,this article proposed an unsupervised feature learning method based on SIFT auto-encoder.In the proposed method,auto-encoder was the basic architecture of the feature learning network,and SIFT descriptors were used as training samples to learn local feature extractors to mine latent feature patterns contained in remote sensing images.In contrast to pixel-based auto-encoder,SIFT auto-encoder has less parameters,lower feature dimension,easier feature extraction process,and better retrieval performance.The proposed method used unlabeled data to learn image features,which solved the problem of image feature learning without labeled data and improved the performance of handcrafted features and pixel-based auto-encoder.(2)Low-dimensional convolutional neural network feature learning and retrieval method for remote sensing image.SIFT auto-encoder improved the performance of handcrafted features,but the improvement was small when compared with some handcrafted feature due to the limited feature learning ability of shallow model.Convolutional neural network often consists of dozens and even hundreds of layers and is able to learn higher-level image features to improve retrieval performance,but a large volume of labeled data is required to train a successful convolutional neural network which is however rare in remote sensing field.In addition,the features extracted by convolutional neural network are usually thousands of dimensions,while high dimensional features require more storage cost and similarity matching time,resulting in low retrieval efficiency.To solve this problem,this article first extracted fully-connected and local convolutional features of pre-trained convolutional neural networks via transfer learning to perform image retrieval,and then proposed a low-dimensional convolutional neural network consisting of convolutional layers and multi-layer perceptron for large-scale image retrieval.The proposed low-dimensional convolutional neural network has less parameters,lower feature dimension and better retrieval performance when compared with the pre-trained networks.The proposed method used limited volume of labeled data for training,which solved the problem of feature learning based on convolutional neural network with limited labeled data and further improved the performance of traditional handcrafted features and SIFT auto-encoder.(3)Multi-label retrieval method for remote sensing image.Most of the current image retrieval approaches are single-label based methods,i.e.,each image is usually described by a single class,and the characteristics are that each image in database is represented by the main semantic class.Moreover,for each query,the classes contained in each returned image is unknown.In fact,remote sensing image scenes are very complex,which makes it is difficult to accurately and effectively represent remote sensing images with only a single label.In addition,single label cannot provide rich semantic information from the perspective of scene understanding,and thus is not able to meet users' fine retrieval requirements.To solve this problem,this article proposed a multi-label image retrieval method based on fully convolutional network.In the proposed method,a fully convolutional network was first trained to conduct semantic segmentation using dense labeling image dataset,which was then used for remote sensing image multi-label analysis and region convolutional feature extraction.Finally,multi-label remote sensing image retrieval was performed based on multi-label vectors and single-and multi-scale region convolutional features.The proposed multilabel image retrieval method can represent image content more accurately since it is able to mine rich semantic information contained in each image via multi-label analysis.In contrast to single-label based methods,the proposed multi-label method is more appropriate for complex remote sensing images with high overlapped classes.