High-Resolution Remote Sensing Image Retrieval Based On Fusion And Pool Ing Of Transfer Features From Convolutional Neural Network

Due to the strong ability to identify object on the ground,high-resolution remote sensing images are widely used in various fields,such as surveying,environment,ecology,and so on.With the increasing number of high-resolution remote sensing images,it becomes an important research topic to retrieve images or targets of interest from remote sensing datasets in the field of remote sensing images.High-resolution remote sensing images contain abundant and complex visual contents,and have rich detailed information.It is important to extract powerful features to represent the complex contents of high-resolution remote sensing images to improve the retrieval performance.Convolutional Neural Network(CNN)can learn image features automatically,and is suitable to deal with high-resolution remote sensing images with complex content.However,due to the small scale of the currently public remote sensing dataset and the difficulty of setting accurate labels to describe the complex contents of the image,it is hard to fully train the parameters of CNN,thus affecting retrieval performance.According to the good transfer learning ability of CNN,this thesis studies to transfer the CNN pre-trained on large-scale dataset for highresolution remote sensing image retrieval.For different sizes of input images,the transfer features are improved from feature fusion,pooling,coding,dimensionality reduction,and fine-tuning.Then the image retrieval performance is improved.The main research work is as follows:(1)The transfer features from different layers of various pre-trained CNNs are investigated for high-resolution remote sensing image retrieval.The parameters of various pre-trained CNNs are transferred for high-resolution remote sensing images,and fully-connected features and convolutional features are extracted for image retrieval.The experimental results show that the transfer features can significantly improve retrieval performance compared with hand-crafted features,and the retrieval performance of fully-connected features is better than that of convolutional features.(2)A feature fusion method based on weight distribution is proposed to improve the transfer feature representation.It is difficult to describe the complex contents of high-resolution remote sensing images completely by a single feature.Thus,the transfer feature representation is improved by fusing different types of features,including the fusion of different transfer features,and the fusion of transfer features with hand-crafted features.The experimental results show that the fusion features are superior to the single feature,and the retrieval results of the fusion features are improved significantly when the weight of the hand-crafted feature is about 0.2.(3)A small region pooling and bag of visual words are proposed to aggregate convolutional features.The small region pooling is proposed based on max pooling and average pooling.By using a small pooling region,the pooling method extracts multiple patch features of an image to represent the detailed information without the need to refeed multiple inputs to the network.The experimental results show that the pooling region size has a great influence on the aggregation features,and the optimal pooling region size for most features is between 50% and 80% of the feature map size.Compared with bag of visual words,the small region pooling method is more suitable to aggregate convolutional features.(4)A region-based cascade pooling method is proposed to aggregate convolutional features.The region-based cascade pooling method first adopts max pooling whose pooling region size is smaller than the feature map size.Then the most salient features are extracted to form the max-pooled feature maps.For these maxpooled feature maps,the average pooling method is employed and the pooling region size is not larger than the max-pooled feature map size.The average pooling treats the salient features equally,which avoid losing the discriminative information.In addition,CNN is fine-tuned to improve feature representation.In the cascading pooling method,the image only needs to be input once to extract multiple patch features,and it combines the advantages of max pooling and average pooling.The extracted features are suitable to represent the complex contents of high-resolution remote sensing images.(5)A method based on combination and pooling is proposed to fuse high-level features from different CNNs.In the combination and pooling method,the high-level features are treated as special convolutional features,so the high-level outputs from different inputs can be obtained,and the three-dimensional tensor of the outputs are retained.Then the high-level outputs of different CNNs are combined into a larger feature map.Finally,the max pooling and average pooling methods are adopted for compression based on the combined feature map.Moreover,PCA dimensionality reduction and fine-tuning methods are adopted to improve the fused feature representation.Experiments show that the retrieval precision of the fused high-level feature of VGG16 and GoogLeNet is 5.98% and 8.79% higher than that of the single feature.In summary,this thesis proposes feature fusion(including weight distribution and combination and pooling),pooling(including a small region pooling and a regionbased cascade pooling),coding with bag of visual words,dimensionality reduction,and fine-tuning for improvement according to the different characteristics of fullyconnected features and convolutional features.Then the image retrieval performance is improved by fully mining the feature information hidden in CNN that is more suitable for high-resolution remote sensing images.