| With the rapid development of visual technology,the information obtained from different sensors is becoming more and more diverse.Image fusion technology can fuse a variety of image information from different sensors in a specific scene to obtain a fused image containing complementary multi-source information,which can help people better analyze and process complex information.Among them,infrared and visible image fusion is an important direction in the image fusion task.This task combines infrared images reflecting thermal radiation information with visible images containing detailed information to obtain a fused image that highlights infrared targets and has more detailed background information.Early image fusion methods were mainly based on technologies in the field of signal processing,including deconstructing the source image through multi-scale transformation and sparse representation.The representation coefficients of the source image are operated and reverted to the fused image.However,processing images in traditional ways can only obtain the shallow representations,which do not perform well on complex scene fusion tasks.Besides,multi-scale transformation is to convert the image to the frequency domain for calculation and the sparse representation needs to learn an over-complete dictionary first.The former has complex calculation steps,and the latter makes the overall operation efficiency of the algorithm low.Then the application of deep learning in computer vision has found a new direction for image fusion.The early image fusion method based on large scale pretrained network is the first attempt of fusion task in deep learning,and then the method based on auto-encoder network further expands the applicability of fusion task.At present,the image fusion algorithm based on end-to-end network is in excellent performance on various fusion tasks.However,the image fusion tasks based on deep learning also face the following problems: how to obtain more effective image features and how to implement a more suitable fusion strategy.In response to the above problems,this paper explores image decomposition and fusion based on deep learning,and studies the application of deep learning algorithms in image decomposition and fusion tasks.The main research contents of this paper include the following aspects:(1)An image fusion method based on feature mutual mapping and multi-scale encoderdecoer is proposed.The method sets different encoder branches for infrared and visible images.The dual-branch encoder maps different source images to different feature spaces to obtain multi-scale features that retain their own modal information.Inspired by the self-attention mechanism,this paper propose a feature mutual mapping fusion module to find the global correlation between two source images.In the fusion module based on feature mutual mapping,the network can adaptively learn fusion strategies during the training process.The aggregation operation of the decoder on features of different scales enables fusion to be achieved at different granularities.Finally,the whole process realizes an end-to-end adaptive fusion method.Our experiments on infrared and visible image fusion tasks and their public datasets demonstrate the effectiveness of the method.(2)An infrared and visible light image fusion method based on a lightweight Transformer module and a generative adversarial network is proposed.In order to focus on the global information in the fusion process,this method combines the Transformer structure to learn fusion relations.This paper propose two Transformer modules that focus on spatial and channel fusion relationships respectively.And the proposed Transformer module is combined into a fusion relation learning module,which is deeply applied to fusion tasks.Adversarial learning enhances the characteristics of different modalities for the fusion image from the feature level in the training process.This is the first attempt to deeply combine Transformer and adversarial learning in image fusion tasks.The method has also achieved good results in subjective and objective evaluations on public datasets.(3)A photorealistic style transfer method based on U-shaped network and multi-layer feature aggregation is proposed.Based on the first work,this paper try to extend the designed encoder-decoder network to similar image tasks.The style transfer task can be viewed as a combined process after decomposing an image into content parts and style parts.In this method,the multi-branch encoder consists of multiple down-sampling modules including dense-blocks,which can fully capture image details while acquiring multi-scale image information.The style transfer module includes adaptive instance normalization(Ada IN)and multi-layer feature aggregation module,which can enrich multi-scale features and enhance the expressiveness of the stylized results.Additionally,this paper design a U-shaped network where upsampling modules and skip connections are used for decoder reconstruction and output stylized images.The experimental results show that the method effectively preserves the original content and structure while improving the image style information,and improves the realistic stylization effect. |
