Sketch-Based 3D Shape Reconstruction

3D reconstruction has long been an important fundamental task in computer vision.3D models have been widely applied in film and television production,architectural design,geological modeling etc.due to the ability to realistically present the shape,size,color and other properties of objects.In the information era,the 3D reconstruction task has become increasingly prominent as the demand for 3D models gradually increases.Sketching is one of the most prevalent design behaviors and is an essential first step in the design process.Professional designers usually use sketches to complete the initial draft and subsequent design processes including refinements and technical implementation are based on sketches.Sketch-based 3D model reconstruction has a high practical application value and has become a top research topic.Traditional sketch-based 3D model reconstruction tasks usually require a large amount of annotation information or user interaction,which is a cumbersome and time-consuming process and difficult to be popularized on a large scale.Benefiting from the rapid development of deep learning and the improvement of computational capability,a growing number of researchers are using deep learning to accomplish 3D reconstruction tasks.However,sketchbased 3D model reconstruction tasks face three main challenges in practicality:(1)the foreground/background ambiguity resulting from the sparsity of sketches.Sketches contain only white background and black strokes,lacking both color and texture information,leading to the difficulties for computers to distinguish foreground/background in sketches;(2)Style differences of sketches.The sketches of each individual have vari ous styles of expression due to their different drawing skills,drawing tools and drawing surfaces;(3)The viewpoint deviations of sketches.In real creation scenarios,even if sketches are requested to be drawn from a specific viewpoint,there is always a viewpoint bias in sketches due to perceptual errors or mechanical errors.In this thesis,we study sketch-based 3D reconstruction without extensive annotation and user interaction to satisfy the actual needs of related creative industries.This thesis achieves innovative research results in the following three aspects:(1)A sketch-based multi-view prediction algorithm for 3D models is proposed,which converts the 3D reconstruction task into an image translation task and effectively alleviates the reconstruction difficulties caused by the huge cross-domain gap between sketches and 3D models.A set of multi-view depth maps and normal maps are predicted from sketches using a specially designed deep adversarial network,and the predicted maps are fused into a 3D model in the following steps.To address the foreground/background ambiguity caused by sketch sparsity,a self-attention module is introduced for the multi-view prediction model and a self-attention loss is proposed,which allows the network to focus not only on local details but also on global structural information during the training stage.For the style difference and viewpoint deviation of sketches,a spatial transform module is employed to process the input sketches and learn the global transform parameters to increase the geometric invariance of the prediction network.Besides,a professional hand-drawn sketch database with paired 3D models is established,which helps to train the prediction network applicable to the real creation environment.Extensive experiments demonstrate the effectiveness of the algorithm and the corresponding module.(2)A 3D model reconstruction framework based on sketch foreground mask prediction is proposed.3D models with multi-view representations usually require a time-consuming view fusion step,so a deep learning framework for directly reconstructing 3D models is investigated.The inherent differences between sketches and colorful images as well as the serious challenges they pose to classical 3D reconstruction algorithms are thoroughly analyzed.A largescale synthetic sketch database is established,containing more than 13,000 3D models and more than 1.8 million sketches,which provides a rich data resource for deep learning.To build this database,sketch stylization methods and viewpoint perturbation strategies are proposed to generate synthetic sketches with various styles and views,which realistically imitate the diversity of human hand-drawn sketches and help neural networks recognize and understand real human hand-drawn sketches.In addition,a framework of the 3D model reconstruction algorithm based on sketch foreground mask prediction is proposed to predict the binary foreground mask of sketches with or without a small number of sparse labels.The mask can be used as auxiliary information to participate in the subsequent 3D reconstruction process together with the sketch,effectively solving the foreground/background ambiguity problem caused by the sparsity of the sketch.(3)The optimization of the 3D model reconstruction algorithm based on regression loss function is proposed,which effectively solves the problems of style difference and viewpoint deviation of sketches.The existing image-based 3D reconstruction algorithms cannot better handle sketches with diverse styles and viewpoints.Therefore,an universal regression loss function is proposed,which improves the performance of the classical 3D reconstruction algorithm for sketches by constraining the correlation between the feature vectors in the sketch embedding space and the ground-truth 3D model to obtain the embedding space feature vectors that are invariant to the sketch style and viewpoint.And dedicated regression loss functions are designed for the two classical 3D representations of point clouds and signed distance fields(implicit functions).Meanwhile,by combining the regression loss function with the embedding space decomposition method,view-related features and view-independent features are obtained to further improve the reconstruction accuracy.The experimental results demonstrate that the method can effectively improve the robustness and accuracy of existing 3D reconstruction algorithms on sketch tasks as well as the ability to separate different feature information in the embedding space.