Multi-spectral Image Fusion Study Based On Few-shot Instance Segmentation

With the rapid development of sensor technology,multi-source data fusion is becoming increasingly important in various fields.As a research branch of data fusion,multispectral image fusion is not only widely concerned in the field of modern industry and agriculture,but also can be further applied to real scenes such as image tracking monitoring and augmented reality.Thanks to the rise of artificial intelligence and deep learning,multi-spectral image fusion technology based on deep learning applies the idea of multimodality.Due to a series of advantages such as wide versatility,good fusion performance and flexible deployment,it has gradually replaced the traditional multi-spectral image fusion method.However,although deep learning-based image fusion technology has made considerable achievements,there are still many problems in related research.First of all,many existing deep learning method is in view of the fusion image as a whole,although the image is decomposed into the information-rich area and background area by extracting significant area,and apply different fusion strategies to realize image fusion,but due to the influence of extraction of significant figure performance,these methods still can cause certain details of the loss and redundancy.There is no model combining visual task and image fusion to realize image key region fusion.Secondly,due to security,confidentiality,physical limitations and other factors,the application scenarios of the current public multispectral image datasets are relatively limited.It is difficult to collect multi-spectral data in industrial scenarios,especially in power scenarios,and the lack of model training samples will limit the fusion performance.Although there are some researches on multi-spectral image fusion for power equipment,the fusion image cannot effectively highlight the key area information due to the complexity and density of power equipment contour.Aiming at the above problems,this thesis proposes a multi-spectral image fusion method based on instance segmentation,and gives some improvements and feasible schemes for related technologies.The main work of this thesis is as follows:1.A set of multi-spectral datasets for power equipment has been collected and collated:Starting from the specific application and demand of multi-spectral image fusion in the electric power field,this thesis collected and sorted out the common power equipment that is prone to thermal failure in the power grid substation.In the outdoor natural light and indoor scenes,the visible and infrared image datasets of the corresponding power equipment were photographed,collected and sorted out to form the multi-spectral dataset of the power equipment,and all the power equipment were manually labeled at the pixel level.The datasets collected in this thesis effectively extend the existing multi-spectral datasets in the electric power field,and can be applied to various experimental scenes such as image instance segmentation task and multi-spectral image fusion task.2.A single-stage instance segmentation model based on dynamic convolution is proposed: In this thesis,in order to improve the feature expression ability of the instance segmentation model on diverse and complex targets,so that it can generate higher quality multi-spectral fusion images in the follow-up work,an improved channel and spatial attention mechanism module is proposed to embed dynamic convolution,and an adaptive dynamic convolution kernel is generated according to different inputs instead of static convolution.It aims to extract richer feature information using lightweight and high-performance network.Then the multi-scale feature information is further enhanced by feature pyramid network.In this thesis,the effectiveness of the proposed model is verified by ablation experiments and hyper-parameter tuning,and the performance of the proposed model on the problem of power equipment instance segmentation exceeds the performance of the existing common general instance segmentation model on the same task.3.A multi-spectral image fusion model based on instance segmentation is proposed:Aiming at the problem of image information redundancy and image information loss caused by current multi-spectral image fusion methods,a multi-spectral fusion model based on instance segmentation is designed in this thesis.It aims to make full use of the background information of visible image with rich details and clear texture,reduce the loss of the temperature information carried by infrared image during the fusion process,as well as further reduce the interference of background noise on the fusion image,and highlight the temperature information of the key area.The multi-spectral image fusion model is divided into two stages.First,the target object is segmented accurately from the visible image according to the pixel level through the instance segmentation model.Then the corresponding area is intercepted from the aligned infrared image according to the segmentation result,and the background pixel is removed and the visible image is fused.The model can retain the complementary information of the source image to the greatest extent,and generate efficient and high-quality fusion images with lightweight fusion rules.In this thesis,the network structure and training details of the model are described in detail,and the experimental results show that the model can effectively generate high quality fusion images.4.A multi-spectral image fusion model based on few-shot instance segmentation is proposed:In order to expand the application scope of the proposed method,this thesis designs a few-shot instance segmentation model based on reverse learning embedded in multi-spectral image fusion task,aiming at the application scenarios where the data collection is difficult and the manual pre-labeling cost is high.Compared with traditional methods such as data augmentation and transfer learning to improve the experimental performance of small datasets,few-shot learning can achieve the task with very low training cost under the condition of ensuring the model accuracy,and can effectively suppress the over-fitting problem caused by data augmentation.In this thesis,the training strategy of reverse learning is adopted to effectively enhance the learning ability of the few-shot instance segmentation network for new categories by making full use of the feature information of both support set and query set without increasing the reasoning cost.The multispectral fusion model based on few-shot instance segmentation can quickly learn the instance segmentation of new target objects at low cost,and be further used to generate with quality multi-spectral fusion images.In conclusion,the multi-spectral datasets of power equipment collected and labeled in this thesis supplement the multi-spectral datasets in the industrial field,especially in the field of electric power,and can be widely used in such tasks as instance segmentation and image fusion.The multi-spectral image fusion method based on instance segmentation proposed in this thesis can fully retain the complementary information of source images,and the improvement of the attention mechanism and the introduction of dynamic convolution in the instance segmentation model can effectively improve the segmentation accuracy of the model for multi-scale objects.In view of the difficult sample collection and high labeling cost,this thesis proposes a multi-spectral image fusion model based on few-shot instance segmentation based on the idea of few-shot learning.The few-shot instance segmentation model is trained by reverse learning,which reduces the manual labeling cost during model training and expands the application range of the proposed method.