Research On Structured 3D Scenes Semantic Segmentation

Semantic segmentation of structured 3D scenes,as the basis of a variety of computer vision tasks,plays an important role in the fields of scene analysis,human-computer interaction,augmented reality,and autonomous driving.In recent years,deep neural networks have greatly improved the results of semantic segmentation of structured 3D scenes.However,there still exists some problems of structured 3D scene semantic segmentation in terms of prediction accuracy,computational complexity,and dependence on semantic labels.In this paper,we focus on the semantic segmentation of structured 3D scenes based on the 3D data obtained by the stereo images and the 3D point cloud data collected by the sensors such as LiDAR(light detection and ranging)and Kinect.Based on the stereo camera,by gathering the color,texture,geometry and disparity information,this paper proposes a novel curb descriptor to depict the characteristics of the curb region,thereby improving the accuracy of curb segmentation.Based on the results of curb segmentation,we further propose a curb-based road and sidewalk segmentation method.For the 3D point cloud data collected by sensors such as LiDAR and Kinect,this paper introduces deep learning technology to improve the semantic segmentation of structured 3D scenes.Considering the complexity and high cost of the semantic annotation on 3D point clouds,this paper further proposes a semisupervised semantic segmentation model for 3D point clouds of structured scenes.In addition,facing the problem of poor generalization ability of segmentation models in practical applications,this paper further proposes an unsupervised domain adaptation semantic segmentation model.The main research contents of this paper include:(1)Aiming at the problem of curb segmentation,a small target in structured road scenes,this paper proposes a curb segmentation method based on stereo cameras,and further proposes the road and sidewalk segmentation methods based on curb segmentation.By fusing various information of color,texture,geometry,and disparity,this paper proposes a discriminative curb descriptor to assist curb segmentation.Based on the curb detection,we construct a cost map and find the two shortest paths from the vanishing point to the bottom row of the image.The curb information encourages the pixels on road boundaries to have lower costs,so as to boost the segmentation of the road.According to the segmentation result of the road and the geometric features of the 3D scene,we sample the seed points of sidewalk area,and employ the region-growing strategy to conduct sidewalk segmentation.Experiments demonstrate that our method achieves good results on the curb,road and sidewalk segmentation.(2)Considering the problem of how to effectively propagate a wide range of contextual information in 3D point clouds,this paper introduces the non-local spirit to enhance the collection of contextual information in 3D point clouds and improve the inference of semantic labels on the point cloud.However,directly applying the non-local operation on the whole point cloud will cause the high computational complexity and memory occupation due to the large scale of the point cloud.To deal with this problem,we propose a cascaded non-local neural network,named PointNL.By propagating the local geometric features gradually to the global area through the cascaded non-local structure,we improve the model performance and effectively control the computational cost of the network.(3)Semantic annotation of 3D scenes requires a lot of human and material resources.Facing this issue,this paper proposes a semi-supervised semantic segmentation framework,named SSPC-Net,for the structured 3D scenes.This method first adopts an unsupervised superpoint partition approach to divides the 3D point cloud into superpoints and extend the point-level label to the superpoint-level label.Then based on the superpoints we construct a superpoint graph,and based the superpoint graph and establish a graph neural network(GNN)to learn the geometric features of superpoints.To enhance the label propagation of the limited labels,we propose a dynamic label propagation method combined a superpoint dropout strategy to generate high-quality pseudo labels.Besides,in order to make full use of the limited supervision and pseudo labels,we further propose a coupled attention mechanism so that the features of the supervised and extended superpoints can be boosted each other.Experiments on various 3D datasets demonstrate that our semi-supervised segmentation method can achieve good segmentation results with only part of the semantic labels and even surpasses the performance of some fully-supervised semantic segmentation methods.(4)Facing the problem of poor generalization ability of semantic segmentation model in practical applications,this paper further proposes an unsupervised domain adaptation semantic segmentation model,named PCSeg-UDA,for structured 3D scene.To minish the divergence of geometric features between two domains and boost the generalization of the model from the source domain to the target domain,this method dynamically generates pseudo labels on the target domain for self-supervised learning,and proposes a cross-domain spatial correlation mechanism and multi-level cycle consistency constraint.The cross-domain spatial correlation mechanism performs feature mapping by establishing connections between domains,so as to enhancing the geometric feature propagation between the two domains.The multi-level cycle consistency constraint encourages the high similarity of the multi-level features of the superpoint pairs on the cross-domain cycle path,thereby reducing the feature offset between two domains.Experiments on the structured indoor and outdoor 3D scene datasets prove the generalization and effectiveness of the UDA semantic segmentation model.