High-Precision Three-Dimensional Human Body Reconstruction And Its Application In Virtual Try-on

Three-dimensional(3D)human body and virtual garment have always been the important research directions in computer graphics and computer vision,which have been widely used in the fields of film,3D animation,games,apparel design,virtual try-on,and E-commerce,etc.Currently,the modeling of the human body and garment mainly relies on 3D scanning or manually designed by professional designers,which is hard to be applied on many real-time applications such as online virtual try-on due to its high cost and inefficiency.Recently,some learning-based methods have been proposed to generate a completed human body efficiently,however,most of them focus on the pose estimation rather than the anthropometric accuracy of the reconstructed human body,and a highprecision human body is a basis for many apparel-related applications such as apparel design,size recommendation,and virtual try-on.The purpose of this dissertation is to investigate the high-precision 3D human body reconstruction and its application in virtual try-on.First of all,to quantify and validate the anthropometric accuracy of the reconstructed body shapes,this dissertation proposes a non-rigid mesh segmentation algorithm for body shape,and then develops an automatic pipeline for 3D human body landmarks extraction and measurement based on the segmentation results.Subsequently,this dissertation proposes two high-precision 3D human body reconstruction methods based on traditional geometric optimization and graph convolution network,from different perspectives.And finally,we extend the two 3D human reconstruction methods and design a novel network architecture for 3D garment retargeting.Specifically,the main contents and contributions of this dissertation are as follows:1)Non-rigid mesh segmentation in the spectral domain.To extract the human body landmarks precisely,this dissertation firstly proposes a non-rigid mesh segmentation algorithm for the 3D human body based on the combinatorial descriptor in the spectral domain.Specifically,we first design a linear Laplacian operator that involves the mesh saliency,and then construct a combinatorial descriptor based on the global and local features that are calculated according to the saliency Laplacian spectrum.During the construction procedure,we also propose a face-level filter,which not only effectively reduces the noise in the saliency detection,but also make the segmentation boundaries to be located at the joint areas.Furthermore,the proposed method adopts an automatic mechanism to determine the number of segments.For some complex shapes,this mechanism could provide a variety of reasonable segment styles as well,which improves the practicability of the algorithm.Extensive experiments demonstrate that the proposed method is effective and efficient for non-rigid mesh segmentation,and is also robust to geometric and topological noises.The segmentation results are comparable with other state-of-the-art algorithms,including some learning-based methods.In the end,we exhibit a potential application for skeleton extraction and skinning as well.2)3D human body landmarks extraction and measurement.Based on the results of 3D body segmentation,this dissertation proposes an automatic pipeline for 3D human body landmarks extraction and measurement,which not only offers the algorithmic fundament for evaluating the anthropometric accuracy of the reconstructed body shapes,but the extracted landmarks can also be used as a significant input for the subsequent reconstruction algorithm.Specifically,we utilize various technologies such as K-Nearest Neighbors,regression,skeleton extraction,shortest path algorithm,and local loop-cutting to extract 22 landmarks from the 3D body directly,which can satisfy various measurement requirements in the field of clothing.We also compare with another measurement algorithm quantitatively.Not only does it achieve the best in multiple measurement sizes,but the anthropometric accuracy also satisfies the error requirements specified in GB/T23698-2009 “General requirements for 3D scanning anthropometric methodologies”.Additionally,we further optimize the 3D body segmentation according to the landmarks extracted.The segmentation boundaries can be located at the landmarks precisely,and the boundaries are much smoother.3)Structure-consistent 3D body reconstruction based on geometric optimization.This dissertation employs the optimization technology to achieve precise 3D human body reconstruction.All the reconstructed bodies share identical topology and possess the real high-frequency details.Furthermore,the landmark indices among various human bodies are the same as well.The proposed method essentially deforms a template into the target body mesh precisely,and the deformation is formulated as an optimization problem with the sparse correspondence as constraints.We also elaborately design the iterative style,objective function,and related parameters for the optimization.Besides,to improve the practicability,we do not impose too many limitations on the input target body.The target human mesh could exist boundaries and holes,and even be a 2D Non-Manifold.Moreover,the proposed method can be applied to various body postures.4)3D body reconstruction based on Anthropometric Graph Convolutional Network(Anthropometric GCN).This dissertation proposes a non-parametric method for precisely reconstructing a 3D human body from mask image(s)and a few anthropometric measurements based on Graph Convolution Network(GCN).The proposed method avoids heavy dependence on a particular parametric body model and can regress mesh vertices directly and explicitly,which is significantly more comfortable using a GCN than a typical convolution neural network.By incorporating the anthropometric measurements into the developed Anthropometric GCN and supplemented by the corresponding loss function,the reconstruction accuracy is greatly improved in terms of anthropometrics,and the reconstruction process is more controllable.We also demonstrate that the proposed network possesses the capability to reconstruct a plausible 3D body from a single-view mask image.Additionally,the proposed method can be effortless extended to a parametric method by appending a Multiplayer Perception to regress the parametric space of a specific parametric human model to achieve parametric 3D reconstruction.Extensive experimental analysis and comparison demonstrate that the proposed Anthropometric GCN itself is very useful in improving the reconstruction accuracy.Finally,we explore the application for anthropometric body design as well.5)A 3D garment retargeting network for virtual try-on.Virtual try-on is one of the important applications of the 3D human body.It is an active topic to wear a virtual garment on a variety of 3D bodies plausibly.According to the proposed 3D human body reconstruction methods,this dissertation firstly designs a simple and effective garment retargeting algorithm based on geometric optimization,and utilizes this method to generate part of the training data.Subsequently,this dissertation proposes a double branches graph convolutional network for 3D garment retargeting by extending the proposed Anthropometric GCN.The retargeting network takes body mask image,anthropometric measurements,and sketches including garment folds and wrinkles as inputs,and employs joint training to reconstruct a non-penetrating dressed body.The garment and body shapes are represented by separated meshes.The proposed method can generate plausible garments by learning from a physics-based dataset.The reconstruction efficiency is greatly improved compared with the traditional physics-based simulation,which can satisfy the real-time application.Extensive experimental results demonstrate that the proposed method is not only comparable with other state-of-the-art learningbased methods but also improves body precision.