Research On Remote Sensing Image Fusion Based On Convolutional Neural Network

In recent years,satellite remote sensing technology has played an increasingly important role in the field of territorial space monitoring,providing solid data guarantee and technical support for it.Satellite remote sensing technology has helped humans realize multi-dimensional dynamic monitoring of territorial space.With the deepening application of remote sensing technology in the field of territorial space monitoring,higher requirements for the spatial resolution and spectral resolution of remote sensing images have been put forward.However,due to the limitations of satellite sensor technology,it is difficult to obtain remote sensing images with both high spatial resolution and high spectral resolution by using a single satellite sensor.Therefore,most optical remote sensing satellites usually provide two different types of images,such as panchromatic(PAN)images with high spatial resolution and multispectral(MS)images with low spatial resolution.Remote sensing image fusion aims to fuse PAN images with MS images to obtain MS images with high spatial resolution.Therefore,it can provide high-quality remote sensing images for subsequent remote sensing tasks such as image classification and target recognition.Remote sensing image fusion technology has become one of the research hotspots in the field of remote sensing,and has very important theoretical significance and practical application value.At present,deep learning technology,represented by convolutional neural network(CNN),has been widely used in the field of image processing and computer vision.CNN can automatically extract a variety of feature information from remote sensing images.Besides,it not only has strong feature representation ability,but also has strong robustness and fault tolerance.CNN has developed rapidly in the field of remote sensing image fusion,but it still faces many challenges.For example,features it extracts are not complete,and the difference and complementarity of multi-scale feature information is not considered enough.Therefore,to improve the quality of the fusion image,the current problem that needs to be addressed is how to use CNN to extract more effective feature information from complex remote sensing images.To solve the above problems,the thesis focuses on the study of feature extraction.Based on CNN,the thesis studies how to construct an effective fusion framework from the perspectives of single-scale,multi-scale,and multi-resolution multi-scale.To improve the quality of the fusion image,the framework is used to extract more abundant features,which is conducive to achieving a balance between spectral and spatial quality.The main work and contributions of the thesis are as follows:(1)A remote sensing image fusion method based on progressive cascade deep residual network is presented.In the image fusion process,due to the inconsistency between the original MS image and the PAN image size,the MS image is usually directly upsampled to the PAN image size.However,it is relatively difficult for vanilla CNN to perform non-linear feature mapping learning in the high-dimensional feature space.Therefore,it is easy to cause the loss of some precise high-frequency details,resulting that the features extracted are not complete.To solve this problem,a progressive cascade deep residual network is proposed for remote sensing image fusion,which is mainly composed of two residual subnetworks.The first residual sub-network is used for feature extraction in the low-dimensional feature space to obtain a preliminary fusion result,which will be sent to the second residual sub-network.The second residual sub-network is constructed to extract more refined feature information in the high-dimensional feature space.In addition,to avoid the over-smoothing problem in the fused image,a new joint loss function is defined to capture the detail difference between the fusion image and the reference image,which is beneficial to improve the spatial resolution of the fusion image.Finally,to avoid checkerboard artifacts in the fused image,the resize-convolution method is used to replace the transpose convolution to perform the upsampling operation.Many experiments on the simulated and real data of Pléiades,IKONOS and World View-3 satellites are performed.Experimental results show that our method can effectively improve the quality of the fusion image.Compared with other mainstream remote sensing image fusion methods,our method performs well both in subjective and objective evaluation.(2)A remote sensing image fusion method based on multi-scale and multi-distillation dilated network is presented.The existing remote sensing image fusion methods based on multi-scale CNN only integrate the multi-scale features simply after extracting them,without fully considering the difference and complementarity between different scale features.It is easy to cause a lot of fine spatial information to be lost,which affects the fusion performance of remote sensing images.Therefore,according to the characteristics of difference and complementarity between multi-scale feature information,a remote sensing image fusion method based on multi-scale and multi-distillation dilated network is presented to further improve the fusion performance of remote sensing images.Firstly,a clique structure-based multi-scale dilated block(CSMDB)is proposed.The clique structure is used to help the network realize the full interaction of multi-scale feature information,which can extract richer spatial details from the source image to improve the information flow effectively.Then,a multi-distillation residual information block(MRIB)is constructed.In MRIB,the feature map with multi-scale feature information is distilled multiple times to obtain residual information.Therefore,the residual information can help the network obtain more refined multi-scale spatial information to further enhance the learning ability of the network.Finally,a feature embedding strategy is designed to embed the feature information extracted from a series of CSMDBs into MRIBs for effective fusion,to improve the reusability of features.A series of experiments were carried out on the simulated and real data of the three satellites Pléiades,IKONOS and World View-3.Experimental results show that our method is significantly better than other mainstream remote sensing image fusion methods in terms of balancing spectral and spatial quality.(3)A remote sensing image fusion method based on dual-stream CNN with residual information enhancement is presented.The existing remote sensing image fusion methods based on multi-scale CNN only consider multi-scale feature extraction at a single resolution,ignoring the complementarity of multi-scale feature information at different resolutions.As a result,part of the information in the fusion result is lost.To solve above problem,a remote sensing image fusion method based on dual-stream CNN with residual information enhancement is proposed.Firstly,to improve the reusability of feature information,a dual-stream information complementary block(DSICB)is constructed to extract rich feature information of different scales at different resolutions,and make them interact fully.Then,to further improve the feature extraction ability of the network,a residual information enhancement strategy is designed to effectively integrate the residual information of different levels.A series of experiments were carried out based on the simulated and real data of multiple satellites.Experimental results show that our method can effectively boost the fusion performance,and is better than other mainstream remote sensing image fusion methods in both subjective and objective evaluation.