| As an important computer vision task,salient object detection can effectively detect salient targets and regions in images or videos,and as a pre-processing step for many computer vision tasks,the accuracy of this task is critical.The salient object detection based on RGB images is easily affected by insufficient illumination and background clutter,and the modal information features are single.Combining depth images and thermal images for salient object detection using multi-modal information features can effectively solve the problem of low scene adaptation for saliency detection.At present,multi-modal salient object detection has achieved a good development,but there are differences between different modalities,how to fully exploit multimodal information and fusion of different modal features still need to be studied in depth.To address the above problems,this paper constructs two efficient multi-modal salient detectors for combining RGB images with thermal images and RGB images with depth images,respectively,and conducts an in-depth exploration of the multi-modal interaction fusion process.The main research is as follows:(1)The theoretical knowledge and development status of existing salient object detection algorithms are studied,and the advantages,disadvantages and current main problems of various methods are analyzed.For the phenomenon of lack of salient object detection video dataset,the depth estimation network is used to generate the depth map corresponding to the existing RGB video dataset and construct the RGBD video dataset.(2)The RGBT image salient object detection model with multi-modal feature aggregation is proposed.The differences and complementarities between RGB and thermal modal features are explored,and an attention-guided module for high-level semantic features and a fast fusion module for low-level detailed features are designed for the fusion between the two modalities.Experiments show that the effective modal interaction of the model can improve the discrimination of salient targets,and the mean values of S-measure,F-measure and MAE metrics on VT5000,VT1000 and VT821 datasets reach 0.900,0.824 and 0.030,respectively.(3)A multi-layer enhanced RGBD video salient object detection model is proposed.The perception of target temporal information is achieved by accurately capturing the correlation of salient targets in continuous images.The RGB modal features are enhanced using information such as geometric shape and spatial location in the depth modality,and the discrimination of salient regions is guided in the decoding stage.Experiments show that the model facilitates the expression and fusion of modalities and improves the effectiveness of multi-modal salient object detection,and the average values of S-measure,F-measure and MAE metrics on the DAVIS and DAVSOD datasets reach 0.806,0.680 and 0.057,respectively. |
