Research On Methods For Pixel-level Infrared And Visible Image Fusion

Infrared and visible image fusion aims to describe the same scene from different aspects by combining complementary information of multi-modality images.Infrared and visible sensors have different imaging mechanism.Infrared sensor is sensitive to the heat source and can capture thermal radiations that emitted by objects.Therefore infrared image can highlight the regions of heat source and can work in poor lighting conditions or disguise.However,infrared image has little texture information due to their low spatial resolution.In contrast,visible image has high spatial resolution and contains abundant texture information.Therefore,infrared and visible image fusion can complement advantages to provide more information for different tasks.Infrared and visible image fusion has been widely used in object recognition and tracking,perimeter protection and malpractice appraisement.Besides,it also has been applied in border management,traffic management,anti-terrorist,and other public security tasks.Generative Adversarial Networks(GAN)can extract the deep features automatically and have became a hot research topic.However,most existing GAN-based infrared and visible image fusion methods only designed one discriminator to force generative image to capture gradients that existing in visible image,which leads to the meaningful information is not keeped sufficiently.Besides,the existing GAN based infrared and visible image fusion methods cannot perceive the most discriminative regions,and hence fail to highlight the typical parts existing in infrared and visible images.To this end,we proposed several GAN-based infrared and visible image fusion methods.The contributions of this paper are summarized as follows:1.Infrared and visible image fusion using dual discriminators generative adversarial networksThe existing GAN-based methods only design one discriminator to force the fused result to complement gradient information from visible image,which may lose some detail information that existing in infrared image and omit some texture information that existing in visible image.To this end,we propose an end-to-end dual discriminators Wasserstein generative adversarial network,termed as D2 WGAN,a framework that extends GAN to dual discriminators.In D2 WGAN,the fused image can keep pixel intensity and details of infrared image by the first discriminator,and capture rich texture information of visible image by the second discriminator.In addition,to im-prove the performance of D2 WGAN,we employ the GAN with Wasserstein distance.More-over,in order to make the fused image keep more details from visible image in texture feature domain,we define a novel LBP(local binary pattern)loss.The extensive qualitative and quantitative experiments on two public datasets demonstrate that D2 WGAN can generate better results compared with the other state-of-the-art methods.2.Infrared and Visible Image Fusion using Attention-based Generative Adversarial NetworksThe existing GAN-based methods cannot perceive the discriminative parts of an image,which leads to low contrast and blurry texture information.Therefore,we introduce a multigrained attention module into encoder-decoder network to fuse infrared and visible images(Mg AN-Fuse).The infrared and visible images are encoded by two independent encoder networks due to their diverse modalities.Then,the results of the two encoders are concatenated to calculate the fused result by the decoder.To exploit the features of multi-scale layers fully and force the model focus on the discriminative regions,we integrate attention modules into multi-scale layers of the encoder to obtain multi-grained attention maps,and then the multigrained attention maps are concatenated with the corresponding multi-scale features of decoder network.Furthermore,we design an additional feature loss in the training process to preserve the important features of the visible image,and a dual adversarial architecture is employed to help the model capture enough infrared intensity information and visible details simultaneously.The ablation studies illustrate the validity of multi-grained attention network and feature loss function.Extensive experiments on two infrared and visible image datasets demonstrate that Mg AN-Fuse has better performance than state-of-the-art methods.3.Infrared and Visible Image Fusion using Generative Adversarial Networks with dual attention mechanismThe existing Generative adversarial networks(GAN)based infrared and visible image fusion methods only introduce the attention mechanism in generator,which leads to the target regions and details are preserve insufficiently.To this end,we integrate multi-scale attention mechanism into both generator and discriminator of GAN to fuse infrared and visible images(Attention FGAN).The multi-scale attention mechanism combine the spatial and channel attention,which aims to not only capture comprehensive spatial information to help generator focus on the foreground target information of infrared image and background detail information of visible image,but also constrain the discriminators focus more on the attention regions rather than the whole input image.The generator of Attention FGAN consists of two multiscale attention networks and an image fusion network.Two multiscale attention networks capture the attention maps of infrared and visible images respectively,so that the fusion network can reconstruct the fused image by paying more attention to the typical regions of source images.Besides,two discriminators are adopted to force the fused result keep more intensity and texture information from infrared and visible image respectively.Moreover,to keep more information of attention region from source images,an attention loss function is designed.Finally,the ablation experiments illustrate the effectiveness of the key parts of our method,and extensive qualitative and quantitative experiments on two public datasets demonstrate the advantages and effectiveness of Attention FGAN compared with the other state-of-the-art methods.