Research On Near-Infrared And RGB Image Fusion Algorithm Based On Generative Adversarial Network

Image fusion technology solves the problem of single sensor obtaining single information,and has important application value in classification,recognition,monitoring,and remote sensing.The fusion of near-infrared(NIR)and visible(RGB)images constitutes a crucial task in the realm of multi-modal image fusion.The imaging properties of these two spectra exhibit favorable complementarity,whereby the high transmittance of near-infrared light enhance texture details of the image,whereas visible images present rich colors.The combination of the two can achieve a more effective and compact scene representation.The generative adversarial network in deep learning has strong simulation distribution capability,and can improve the effect of image fusion.Therefore,we propose two methods for NIR and RGB image fusion based on generative adversarial network.To address the problems of uneven texture blending and color distortion of existing algorithms,a two-branch multi-scale cross-fusion algorithm based on GAN is proposed in this thesis.The method divides the source image information into two parts: texture and color.The upper and lower branches of the generator design a full-scale texture fusion network and a color fusion network respectively,and features of the two branches are merged via cross-space attention blocks.Then two Markovian discriminators are utilized to further enhance the detail representation of the fused image.The experimental results demonstrate that the algorithm fully fuses RGB and NIR images,significantly enhances the clarity of target edge,and improves the contrast and richness of color..The above method is a pixel-level fusion,which requires high feature matching and suffers from excessive contrast.On this basis,structure and appearance semantic-guided fusion based on GAN is proposed.The method utilizes pre-trained DINO-Vi T network as semantic prior to guide image fusion from the perspective of structure and appearance.Specifically,we select [CLS] token as the global appearance representation,and construct structural representation using the keys from the trained multi head self-attention.While the generator focuses on semantic information fusion,we adopt two local discriminators to enhance local details on pixel level.The experimental results demonstrate that the algorithm can not only fully integrates the semantic information of RGB and NIR images,but also makes color representation more natural.The above methods study the unsupervised fusion task of NIR and RGB images from the perspective of pixel and semantic,and achieve an effect of generating rich texture and natural color,which can be applied to high-quality imaging in complex scenes.