Semantic And Instance Joint Segmentation Of Building With Oblique Photogrammetry Point Clouds Based On Deep Learning

As a new type of data,oblique photogrammetric point clouds data have received extensive attention because of its rich surface information,low cost,high accuracy,superior efficiency and other advantages and shows a broad application prospect in the aspects of building extraction and building singulation.Compared with two-dimensional images,the difficulty of densely matching point clouds is that when classifying features,the top surface of the features must be classified and the set of elevation points of the features should be divided into corresponding categories simultaneously.However,the dense point cloud generated by dense matching of multi-view images were generally affected by complex and variable surface fluctuation factors in the classification task.When oblique photogrammetry is used to obtain multi-view images,the complete feature photos cannot be obtained due to the shape and the mutual occlusion within land-covers,which may produce several wrong spatial and spectral information points,and causing worse classification results in the traditional 3D point clouds classification algorithms.In recent years,artificial neural networks in deep learning have been widely used in many research fields due to their superior nonlinear expression capabilities.Researchers have introduced deep learning methods in oblique photogrammetric point clouds,so the feature classification in 3D point cloud data is also called semantic segmentation.In the threedimensional point clouds,the building singularity of the point cloud is conceptually similar to the instance segmentation in deep learning.Therefore,this paper attempts to use deep convolutional neural networks to perform semantic segmentation and instance segmentation on oblique photogrammetric point clouds,to improve the frame structure and optimize the part of loss calculation of the network model during the training process.The main work and achievements are as follows:(1)In order to make the oblique photogrammetric point clouds data better fit the calculation and training methods of the fully supervised deep neural network,this research processed the point clouds according to the data format characteristics of the densely matched point clouds,and a series of pre-processing processes such as point clouds segmentation,resampling,3D coordinate conversion,data normalization,point clouds coverage manual tags,and corresponding merger of point clouds data and tag information after pre-processing process was made.Finally,the processing and conversion from the original point clouds data to the point clouds dataset suitable for the network model were completed.(2)In the semantic segmentation section,it pointed out the advantages of deep learning neural networks in semantic segmentation tasks compared to traditional point clouds classification algorithms,and used two deep convolutional neural networks to perform semantic segmentation of buildings for oblique photogrammetric point clouds.Then,after detailing the framework structure and the principles of loss algorithm and training process of the two deep neural networks,the original structure of the two network models was optimized and the algorithm was improved.As a result,the two network models obtained a segmentation accuracy of 93.9% and 98.7% respectively.(3)On the basis of the obtained semantic segmentation results of oblique photogrammetric point clouds buildings,this study used the past common "semantic + instance" paradigm-byparadigm segmentation mode and introduced the previous semantic segmentation results to segment the 3D point clouds with the accuracy of 58.1%.However,in the training process of instance segmentation,we found that the fully supervised deep neural network has the disadvantage that the overall training process spends long time,which results in insufficient segmentation efficiency.Therefore,considering the similarities and differences between semantic segmentation and instance segmentation,we seek common ground while reserving differences,this paper proposed a parallel segmentation model that simultaneously performs oblique photogrammetric point clouds semantic segmentation and instance segmentation and built a network model of joint semantic and instance segmentation.This segmentation mode reduces the network training time greatly and improves segmentation efficiency.Finally,this network model can complete the training of semantic segmentation and instance segmentation at the same time,with an accuracy of 94.1% and 53.2% for semantic segmentation and instance segmentation respectively.