Research On 3D Shape Analysis Based On Convolutional Neural Networks Fusing Multi-view Features

In recent years,with the wide application of 3D modeling softwares and depth sensors,online repositories of 3D shapes with more types and quantities have been explosively growing.How to eftectively analyze,organize and manage these 3D shapes has become an urgent problem to be solved.At present,there have been many research branches in the field of 3D shapes analysis,such as semantic segmentation,classification,shape retrieval,3D reconstruction,and shape completion.3D shape segmentation and classification,which analyze 3D shape at the level of part and category respectively,are the fundamental researches and significantly instructive for many other research branches.In addition,semantic segmentation and classification are also high-level tasks for scene understanding.Therefore,this paper takes the semantic segmentation and classification of 3D shapes as research targets.With the development of deep learning,great changes have taken place in the field of semantic segmentation and classification.Although most deep learning frameworks have achieved very good results in the field of semantic segmentation and classification of 2D images,3D shape segmentation and classification have encountered various difficulties due to the irregular structure of the 3D shape.The method based on the analysis of the rendered projection images has no requirement on the structure and orientation restrictions,and at the same time,it can make full use of the technical advantages accumulated in the field of image analysis to process the 3D shapes'problem.Inspired by this,this paper applies an image-based deep learning framework to the 3D shape's rendered projection images to indirectly complete the semantic segmentation and classification tasks of 3D shapes.Different from the existing methods,the main contributions of this paper are summarized as the following three points:(1)Based on optimal viewpoint selection according to the viewpoint entropy,a convolutional neural network framework fusing multi-view features is designed and implemented,which can achieve the task of semantic segmentation of 3D shapes.We firstly introduce the best viewpoint selection method based on viewpoint entropy into the field of 3D shape segmentation,which reduces the information redundancy between viewpoints.And we design a compact data structure for storing the relationship between the 3D shape's faces and pixels' coordinate in the rendered images from the visual viewpoints among the selected viewpoints and enhance the efficiency of mapping pixels' labels to 3D shape faces.In addition,in the convolutional neural network,multi-view features are fused with single view's feature to enhance the co-analysis of multi-view information at diffierent scales.(2)Based on the multi-layer fully connected framework,a convolutional neural network framework fusing multi-view features is designed and implemented,which can achieve the task of 3D shape classification.We firstly introduce the best viewpoint selection method based on the viewpoint entropy into the field of 3D shape classification,making the selected viewpoint contain more visual information and category characteristics,and reduce the influence of non-uniform distribution of the 3D shape's triangle faces on the best viewpoint selection.Two fully connected layers are used to fuse the multi-view features instead of feeding them into another neural network after down-sampling,which improves the efficiency of training the network.(3)Based on the projection method,semantic segmentation and classification of 3D shapes are unified under the same network framework.Based on the projection method,the semantic segmentation of 3D shapes and the classification of 3D shapes are unified in the same framework,so that the convolutional neural network framework fusing multi-view features can be applied not only to 3D shape segmentation but also to 3D shape classification.We evaluate our method on the benchmarks of 3D shapes segmentation and classification,and it's demonstrated that our optimal viewpoint selection based on viewpoint entropy is effective and fusing multi-view features in the convolutional neural network is essential.It's proved that the convolutional neural network framework fusing multi-view features designed in this paper can process the semantic segmentation and classification of 3D shapes well and it reached the research purpose.