Research On Weighted Fusion Semantic Segmentation Algorithm Of RGB-D Images In Fire Scenes For Fire Fighting Robots

In small and medium-sized fire scenes such as large warehouses with aging wires,factories or old flat floor residential buildings,firefighting robots can replace firefighters to enter the fire scene to extinguish the fire.In the fire environment,the fire robot should not only extinguish the fire,but also find the location of the wounded,which requires the fire robot to have the ability of scene understanding and casualty recognition.However,the identification of the fire scene photographed by ordinary RGB cameras cannot be carried out normally due to problems such as flame and smoke blocking the sight line.Currently,no scholars have studied the data set and network model suitable for this scene.To meet the above needs,this paper develops a firefighting robot with functions such as environment and casualty perception,remote control and fire extinguishing.The multi-modal data set in the fire scene is created for the robot to complete the neural network model training in the special scene,so that it can have stable semantic map creation and casualty detection capabilities.Firstly,by analyzing the application scenarios and functional requirements of fire fighting robots,the hardware platform and control system of fire fighting robots are designed and built according to the corresponding technical indicators,and the prototype of fire fighting robots is developed.Based on the prototype,the key technologies such as scene understanding are studied.Robot hardware and software system is mainly divided into five modules:mobile chassis and drive module,fire and water cannon and control module,camera and image transmission module,remote control communication module and deep learning image processing module.Secondly,in view of the currently unpublished fire scene data set applicable to scene understanding,this paper creates a casualty segmentation data set under fire scene for the use of semantic segmentation network training,including RGB images,depth images and infrared images of the wounded under fire environment.The production and processing of the data set mainly includes image acquisition,pre-processing,manual annotation and image enhancement.Then,aiming at the problem of RGB map failure in fire fighting scenes and taking advantage of the fact that depth map is not interfered by flame smoke,this paper proposes a semantic segmentation network DDNet dominated by depth map,which makes full use of geometric information of depth image in RGB-D data set to analyze and segment indoor scenes.RGB map features and depth map features are extracted by two convolutional neural network branches respectively and fused with a certain weight ratio.In order to further enhance the network generalization ability,this paper first conducts training on ordinary indoor open source data sets to prove the effectiveness and universality of the depth chart-dominated network.The experimental results on NYUv2 and ScanNetv2 open source datasets show that DDNet has better training efect than the semantic segmentation network dominated by RGB maps of the same architecture,which also proves that depth maps are more suitable for segmentation tasks in scene understanding.Finally,the network model designed in this paper is used to train and compare different image modes in the established training environment.The experiment verifies that the infrared image has the best semantic segmentation performance in the fire scene,and the global accuracy can reach 98.7%.In addition,different modal data is trained by weighted fusion,and the experimental results of information fusion show that its performance is better than the semantic segmentation of a single RGB graph or a single depth map.The experiment verifies that the fire fighting robot has a good ability to recognize the wounded in the fire scene.This thesis is supported by Shandong major science and technology innovation project(No.2019JZZY010112),key research and development program(2020JMRH0202),major industrial research projects in Shandong Province for the conversion of old and new kinetic energy(2021-13).