Pan-sharpening Based On Convolutional Neural Network

Pan-sharpening aims to obtain an informative high spatial resolution multi-spectral image by fusing the original multi-spectral(MS)image with panchromatic(PAN)image.We focus on the two main objectives of Pan-sharpening:enhancing the spatial resolution as much as possible while preserving the spectral information,to design an end-to-end deep convolutional network Pan-sharpening model:PanNet.The PanNet model consists of two branches,spectral preservation and spatial resolution enhancement.To keep the spectral information as much as possible,the spectral mapping branch of the PanNet model addes up-sampled multi-spectral images to the network output to directly propagate the spectral information to the reconstructed images.In order to enhance the spatial resolution of multi-spectral images,we train our network parameters in the high-pass domain rather than the image domain,which is learned through the residual network(ResNet)training.Compared with the general deep learning model that directly learns the mapping of low resolution multi-spectral images to high resolution multi-spectral images,this high frequency to high frequency mapping learning allows the model to focus on the improvement of spatial resolution.Moreover,we show that adding up-sampled multi-spectral and using high-pass images help to simplify the learning process.In addition,to improve the performance of our model,we introduce the concept of feature recalibration,and recalibrates the feature maps obtained from the convolution layer in both spatial and feature channel dimension,so that encouraging the network to learn more meaningful feature mappings.Comparing the simulated and real data experiments,our model is significantly superior to the traditional Pan-sharpening based on optimization and deep convolutional neural network in terms of subjective and objective evaluation indexes for spectral preservation and spatial resolution enhancement.In addition,the proposed model has strong generalization performance and can be applied to multispectral remote sensing images from different satellites without parameter tuning.