Infrared And Visible Image Fusion Methods For Intelligent System Perception

In recent years,rapidly developing intelligent system has gradually become a new engine leading scientific and technological progress and driving social development.As the eyes of intelligent systems,visual perception technology largely determines the development level of core functions of the overall system,such as information acquisition and intelligent cognition.In order to realize the all-weather information acquisition and intelligent cognition ability of intelligent system in complex scenes,multi-source image fusion technology has attracted extensive attention.Among them,infrared and visible images,as the most important visual data sources for intelligent system perception,play an irreplaceable role in the realization of high-confidence perception tasks in intelligent systems.Infrared sensor uses thermal radiation for imaging,which can highlight the overall contour characteristics of the object,but there will be blur and halo in the imaging process;Visible light sensor uses light to reflect energy on different scenes and objects for imaging,which can better present the detailed information of scene texture,but it is easy to be affected by objective scene illumination,brightness,bad weather and other factors.Therefore,it is of great significance to give full play to the advantages of infrared images and visible images fusion to improve the perception of intelligent systems.With the rapid development of artificial intelligence,deep learning technology has injected new vitality into the field of infrared and visible image fusion.Due to the obvious differences between infrared and visible images in imaging principle and feature representation,the existing fusion methods based on deep learning rarely effectively learn and integrate these "similarities and differences" features,resulting in the degradation of visual effect and difficult perception of the fused image.In order to solve the above problems,this paper deeply analyzes and explores the infrared and visible image fusion algorithm based on deep learning from three aspects: fusion strategy modeling,fusion feature representation and perception driven learning.The main work is summarized as follows:1.Multi-modality image fusion via bilevel optimization paradigm.The existing fusion strategies do not impose different constraints according to the different characteristics of multimodal images,resulting in poor contrast and loss of information.To solve this problem,the multi-modal image fusion task is modeled by bilevel optimization.According to the characteristics of multi-modal images(mostly composed of base layer and detail layer),implicit priors guided by different layers are designed to restrict the stage fusion results of corresponding layers,which can not only avoid the artifacts in the fusion results,but also highlight the details.In addition,an adaptive weight graph based on deep learning is introduced to replace the manually designed parameters in the model solution.The experimental results show that the visual effect of this method is significantly improved in tasks such as infrared and visible image fusion and medical image fusion,and the objective metrics such as visual information fiedity(VIF)and the sum of correlation differences(SCD)are improved by an average of 5.3% and 6.5% compared with the state-of-the-art fusion methods.2.Network structure design for fused feature representation based on auto-encoder framework.Aiming at the problems that the existing depth network structure is difficult to effectively represent the internal characteristics of infrared and visible images,resulting in the degradation of fusion results,this paper constructs an effective infrared and visible feature representation network structure from manual design to automatic search.First,a coarse-tofine deep architecture is proposed to learn multi-scale feature representations for infrared and visible images,which extracts a more comprehensive feature representation for the subsequent fusion operation,and an edge-guided attention mechanism is designed to achieve high-quality image fusion and restore details while reducing noise.Then,a novel network structure based on neural architecture search is proposed,aiming to automatically learn the intrinsic feature representation of differences from infrared/visible images.Extensive experiments have proved that the proposed method can fully represent the features of different modal images,effectively avoid the loss of information in the extraction of features by the network,and improve the visual effect of the fused images.It has achieved the best results in objective evaluation indicators,such as mutual information(MI)and structural similarity index measurement(SSIM).3.Target-aware infrared and visible image fusion for object detection.Aiming at the problem that the previous infrared and visible image fusion methods only explore and focus on the visual quality of the fused image,but ignore the downstream visual task,a perception task oriented infrared and visible image fusion method is proposed.Firstly,a task driven fusion optimization model is designed,and then it is expanded to a target-aware dual adversarial learning network for fusion and a commonly used detection network.The fusion network has a generator and a double discriminator,which aims to "seek common ground while reserving differences" to learn and use the different features of infrared and visible images.Therefore,the structural information and texture details of the target can be retained in infrared and visible source images respectively.In addition,a four-eye synchronous imaging system is constructed by using infrared and visible optical sensors,and the most comprehensive registered infrared and visible data sets are collected,covering a wide range of application scenarios,such as low illumination,smoke,shielding,camouflage and so on.A large number of experiments show that the proposed method is not only visually attractive,but also has an average detection rate of 10.9% higher than the most advanced fusion methods in various challenging target detection scenes.In addition,a four-eye synchronous imaging system is constructed by using infrared and visible optical sensors,and the most comprehensive infrared and visible data set is collected,which covers a wide range of application scenarios,such as low illumination,smoke,occlusion,camouflage and so on.Extensive experiments on several public datasets and our benchmark demonstrate that our method outputs not only visually appealing fusion but also averagely 10.9% higher detection m AP than the state-of-the-art approaches on various challenging scenarios.