Research On Image Fusion Based On Deep Learning

The images obtained by different modal sensors in the same scene are fused to obtain a fused image containing multi-source information.This type of task is called image fusion task.Among them,the fusion of infrared image and visible light image is an important topic.Infrared images can distinguish the target from the background based on the difference in thermal radiation.Infrared images can work well at all times of day or night and under various weather conditions.In contrast,visible light images can provide texture details with high spatial resolution and clarity in a manner consistent with the human visual system.There are already a large number of mature methods in the field of image fusion,which are roughly divided into two categories.One is to use deep learning network to generate a fusion image in one step.The algorithm inputs infrared images and visible light images,and the network learns and generate the required fusion image adaptively.Another is to use traditional methods or deep networks to decompose the image,convert it to a specific space and fuse its features,and finally convert the fused features to the original image space to obtain the fused image.Using traditional methods to perform image fusion usually requires rigorous design of algorithms,and generally speaking,the fusion process is time-consuming.However,image fusion methods using deep learning appeared late,and most methods did not consider the features of the image and lack a reasonable network structure design.Based on this,inspired by the one-step image fusion of deep learning,a perception-based Generative Adversarial image fusion networks is proposed.At the same time,for the analysis and thinking of image features,a feature-enhanced autoencoder structure is proposed.Finally,based on the general decomposition steps of image fusion,two general methods of image decomposition based on deep learning are proposed.The main contributions of this article are as follows:(1)A new method based on dense blocks and GANs is proposed.And the proposed method directly insert the input image-visible light image in each layer of the entire network.The proposed method use structural similarity and gradient loss functions that are more consistent with perception instead of mean square error loss.After the adversarial training between the generator and the discriminator,a trained end-to-end fusion network –the generator network–is finally obtained.The experiments show that the fused images obtained by the proposed method achieve good score based on multiple evaluation indicators.Further,the fused images have better visual effects in multiple sets of contrasts,which are more satisfying to human visual perception.(2)A novel image fusion method for fusing infrared images and visible light Images is proposed.The backbone of the network is an autoencoder.Different from previous autoencoders,the proposed method enhance the information extraction capability of the encoder and optimize its ability to select the most effective channels in the decoder.First,the proposed network extract the features of the source image during the encoding process.Then,a new effective fusion strategy is designed to fuse these features.Finally,the fused image is reconstructed by the decoder.Compared with the existing fusion methods,the proposed algorithm achieves state-of-the-art performance in both objective assessment and visual quality.(3)A new image fusion method by designing a new structure and a new loss function for a deep learning model is proposed.The backbone network is an autoencoder,in which the encoder has a dual branch structure.The proposed method input infrared images and visible light images to the encoder to extract detailed information and semantic information respectively.The fusion layer fuses two sets of features to get fused features.The decoder reconstructs the fusion features to obtain the fused image.The proposed method have a new loss function to reconstruct the image effectively.Experiments show that the proposed method achieves stateof-the-art performance.(4)Image decomposition is a crucial subject in the field of image processing.It can extract salient features from the source image.A new image decomposition method based on convolutional neural network is proposed.This method can be applied to many image processing tasks.In this paper,the proposed method apply the image decomposition network to the image fusion task.The network input infrared image and visible light image and decompose them into three high-frequency feature images and a low-frequency feature image respectively.The two sets of feature images are fused using a specific fusion strategy to obtain fusion feature images.Finally,the feature images are reconstructed to obtain the fused image.Compared with the state-of-the-art fusion methods,this proposed method has achieved better performance in both subjective and objective evaluation.(5)The existing visual transformer models aim to extract semantic information for highlevel tasks such as classification and detection,distorting the spatial resolution of the input image,without the capacity in reconstructing the input or generating high-resolution image.In this paper,therefore,a Patch Pyramid Transformer(PPT)is proposed to effectively address the above issues.Specifically,the proposed method first design a Patch Transformer to transform the image into a sequence of patches,where transformer encoding is performed for each patch to extract local representations.In addition,a Pyramid Transformer is constructed to effectively extract the non-local information from the entire image.After obtaining a set of multi-scale,multi-dimensional,and multi-angle features of the original image,the image reconstruction network is designed to ensure that the features can be reconstructed into the input image.To validate the effectiveness,the proposed Patch Pyramid Transformer is applied to the image fusion task.The experimental results demonstrate its superior performance against the state-ofthe-art fusion approaches,achieving the best results on several evaluation indicators.