Full-time Pedestrian Detection Based On Adaptive Fusion Of Multispectral Data

Multispectral pedestrian detection has become one of the key technologies for full-time operation and monitoring such as autonomous driving and security defense,and multispectral pedestrian detection technology based on deep learning has also become the top priority in multispectral pedestrian detection research.Although the existing multispectral pedestrian detection technology based on deep learning has made great progress compared with the past,it still has the following shortcomings: 1.Most of the existing multispectral pedestrian detection models directly combine the original visible light image and thermal infrared image.As an input,the thermal infrared image obtained directly by the thermal infrared camera will be interfered by other heat sources or objects on the road that become hot due to light,which will cause the thermal infrared image background to be complex and the object area to be dim and uneven.In addition,since there are fewer pedestrian targets in the road scene than other background targets in the whole image in general,the existing data augmentation methods may cause the problem of the loss of pedestrian targets while enriching the background when used in the field of pedestrian detection.2.In the current multispectral pedestrian detection algorithms,the fusion weights of visible light and thermal infrared modalities are often designed based on the overall illumination intensity of the entire image,ignoring the local shadows.A fusion method that adaptively adjusts the fusion weights according to local information.3.The current multispectral pedestrian detection algorithm is not fast enough when the detection accuracy is reached,and the automatic driving is running fast.We seek to improve the detection speed as much as possible on the premise of ensuring the detection accuracy to achieve the best balance between speed and accuracy.In view of the above problems,this paper studies from the following aspects:1.In view of the problem that the image is not pre-processed before input,it is proposed to use low-pass filtering technology to enhance the thermal infrared image.By embedding the filter chip in the thermal infrared camera,image enhancement is performed at the moment of shooting,and the thermal infrared image obtained can reduce the interference noise caused by the background heat source.In addition,a data enhancement method,Limt-Merge Mix,is proposed.Before the model is trained,the irrelevant background information of the upper and lower regions of the two images is cropped,the remaining images and labels are merged up and down,and the two adjacent images of thermal infrared and visible light are merged.The two adjacent images are used as new training data to train the model,which effectively enriches the background information without losing the pedestrian information to be trained.2.In view of the problem that the image is not pre-processed before input,it is proposed to use low-pass filtering technology to enhance the thermal infrared image.The interference noise caused by the background heat source can be reduced by processing the thermal infrared image using a low-pass filtering method.In addition,a data enhancement method,LimtMerge Mix,is proposed.Before the model is trained,the irrelevant background information of the upper and lower regions of the two images is cropped,the remaining images and labels are merged up and down,and the two adjacent images of thermal infrared and visible light are merged.The two adjacent images are used as new training data to train the model,which effectively enriches the background information without losing the pedestrian information to be trained.3.It is proposed to apply the cross-stage localized design to the YOLO Neck structure to reduce model parameters,reduce model calculation amount,and improve model detection speed while having multi-scale detection capabilities.In the multi-scale fusion network with multi-spectral fusion network structure,a cross-stage localization design is carried out.Compared with the original YOLOv4 model,the improved multi-scale network model can not only enrich the network feature level through the bottom-up feature enhancement path,but also extremely It greatly reduces the number of parameters and calculation of the model.Compared with the model using the original YOLO Neck,the YOLO Neck reduced by CSP reduces the number of floating-point operations by 14.1% and the model size by 23.2%.Experiments on the network model after transforming YOLO Neck show that the detection speed of a single image is 0.034 s,which is 0.058 s faster than the CS-RCNN detection model when detecting a single image.