A Deep Learning Pan-sharpening Method Driven By Domain Knowledge

Pan-sharpening refers to the process of fusing a panchromatic(PAN)image and a multispectral(MS)image for the same scene to obtain a high resolution multispectral(HRMS)image.Pan-sharpening has overcome the physical limitations of a single sensor effectively without upgrading the hardware,which has led Pan-sharpening to become a hot issue and has been widely used in natural resource exploration,environmental monitoring,military reconnaissance,and disaster monitoring.The purpose of Pan-sharpening is to improve the spatial resolution under the premise of no spectral distortion.The MS image has a high spectral resolution but low the spatial resolution and the PAN image has a high spatial resolution but lacks spectral resolution.Since the same scene is imaged,the MS image has a similar image structure to the PAN image,but the two imaging bands are inconsistent,resulting in a large difference in the numerical value.Based on the above knowledge,this paper presents two methods of Pan-sharpening for remote sensing images.The main contents and achievements are as follows:Pan-sharpening with structural consistency ande l1/2 gradient prior is proposed.The main contributions of this method are as follows:1)The spatial structure consistency is achieved by using the difference between the gradient of PAN image and HRMS image,which obtains the spatial structure information from the PAN image.The objective function makes effective use of structural similarity between the PAN image and HRMS image and reduce the difference by introducing gradient information.The sparse characteristic of l1 norm further strengthens the structural similarity.2)A large number of statistical experiments show that the gradient of HRMS images matches the hyper-Laplacian distribution.Therefore,the l1/2 norm is better to characterize this relationship than the commonly used l1 or l2 norm.In addition,an l1-αl2 metric is used to estimate l1/2 gradient prior,which solves the above variational model efficiently.A deep learning pansharpening method driven by domain knowledge called Deep Pan-sharpening is proposed.Different from general deep learning methods which learn the mapping between input image and output image directly,the proposed end-to-end the network takes full account of the Pan-sharpening domain knowledge,that is,improve the spatial resolution under the premise of no spectral distortion:1)The upsampled low resolution multispectral(LRMS)image as a shortcut connection is added directly to the network output so that preserving spectral information as much as possible.2)The residual network is used to learn the mapping of the high frequency structure information of the PAN image and the LRMS image to the high frequency structure information of the HRMS image,On the one hand,the high frequency map improves the spatial resolution of the LRMS image as much as possible,on the other hand,it reduces the spectral distortion caused by the difference of image values.Comparing with the recently proposed Pan-sharpening methods,the above two methods have some advantages both subjectively and objectively.Among them,Deep Pan-sharpening doesn’t need to tune many parameters and is more robust than other network structures.