| With the development of laser sensing and multi-view stereo matching technology,a large number of 3D point cloud data have emerged,laying a solid data foundation for using computers to understand the 3D environment around us.How to intelligently(automatically,efficiently,and reliably)learn and represent these massive point cloud has been a key problem that needs to be solved for 3D understanding(including 3D object recognition,semantic segmentation,and 3D object generation,etc.).However,traditional point cloud feature descriptors are facing two main challenges with respect to these massive 3D point cloud,which are: 1)Due to the characteristics of huge volumes,multiple object classes and scene complexity of 3D point clouds,traditional low-level features are often difficult to characterize the high-level semantic information of the point clouds;2)Traditional low-level features need to be manually designed and extracted for different types of objects,which has a low-level intelligence,and is difficult to adapt to the complex real situation that has uneven density,a lot of noise,local data missing and mutual occasion.Recently,benefiting from the powerful feature learning capabilities,deep learning has drawn extensive attraction in the field of image understanding,and has achieved fruitful results.Considering the fact that traditional man-designed 3D point cloud features cannot describe high-level semantic information and are difficult to adapt to complex real situations,by revisiting the 2D convolutions on regular image,this paper mainly studies how to generalize deep learning in regular domain to irregular 3D point cloud for feature learning,which lays a foundation for the analysis and understanding of 3D environment.Specifically,the research in this paper mainly includes the following four aspects:(1)Multi-scale 3D convolution based semantic segmentation of point cloud.The most direct way to apply convolutional deep network to a point cloud is to voxelize the3 D point cloud space,and then generalize the 2D regular image convolution to the 3D point cloud for feature learning.However,unlike the regular and compact information organization of 2D images,3D point clouds actually only record the surface information of 3D objects,which causes most of the voxels in space to be empty,which brings unnecessary computation to the 3D convolutional network.In fact,convolution can be regarded as a process of weighted-combination of neighboring points.Since closer points always lead to greater spatial relation,this paper proposed to voxelizing the distant neighbor area with coarse scale,so that it can reflect the overall context of the neighborhood,and voxelizing the closer neighbor area with fine scale,so that it can focus on the detailed structure of the point cloud.In this way,this multi-scale 3D convolution not only greatly reduces unnecessary calculation waste,but also improves the stability of feature extraction with respect to noise.(2)Deformable 3D point kernel based geometric feature mining.Focusing on the computational waste caused by the voxelization of point cloud space and the problem that traditional point cloud feature descriptors cannot adapt to complex real situations,we further proposes a point cloud geometric feature learning convolution based on deformable 3D point kernels.Convolution is actually a template matching process,the convolution kernel slides on the signal and responds to the interested patterns.In 3D point cloud,we can directly use 3D points to form a series of "templates" with specific geometric structures,so that to detect and capture the geometric structures in the 3D point cloud by matching these geometric templates with the local neighborhoods of the point cloud.At the same time,in order to adapt to complex real situations,each 3D deformable kernel is composed of a set of learnable 3D points.During the training process,by iteratively adjusting the coordinates of each point,these points can be formed a certain geometric structure and shape(such as planes,corners,etc.).Since the 3D point kernel directly operates on the neighborhood points,it avoids the computational waste caused by the spatial voxelization of the point cloud,while ensuring the effective description of the geometric structure in the point cloud.(3)Attentional convolution based semantic segmentation of point cloud.The parameters of the standard convolution on regular 2D image are actually determined by the relative positions of the neighborhood points.Due to image’s regular neighborhood arrangement,the convolution weight at specific positions(such as the upper-left corner and the lower-right corner)are fixed,and the neighborhood information in the convolution window are finally aggregated.However,this neighborhood position based convolution ignores the feature differences between the neighboring points.For example,when two objects "table" and "chair" are adjacent in space,the features extracted at the junction will be contaminated and cannot be used to accurately segment these objects.Therefore,this paper proposes an attentional convolution that dynamically assigns different attention weights to each neighborhood point by combining their relative position and feature differences,so that the shape of the convolution kernel can be adapted to the structure of objects,which makes the learned features can focus on the current object and then achieve high-precision point cloud semantic segmentation.(4)3D point cloud generation by combining conformal transformation and adversarial autoencoder.3D object generation is a higher goal of intelligent 3D understanding,which needs to create plausible 3D objects according to specified conditions.This can be considered as the inverse process of feature encoding.Current image generation method are mostly based on generative adversarial networks and variational autoencoder.However,directly discretizing 3D point clouds into voxels and generalize image generation network to 3D object not only limits the resolution of the generated 3D object,but also brings huge computation waste.In fact,point cloud only records the surface information of 3D object,which makes the 3D point cloud only constitute a 2D manifold surface embedded in the 3D space.Therefore,this paper first uses a conformal transformation to map the point cloud of a 3D object onto a 2D regular plane,so that the 3D object generation problem is transformed to a problem of generating 2D image.Then the classical image generation network can be referred for generating 3D object’point cloud. |
