Research On Portrait Relief Modeling From Single Image Via Deep Learning

Traditional relief modeling is usually handmade by artists,which not only requires professional skills,but also takes time and effort.In recent years,researchers have proposed a number of methods for relief modeling through numerical optimization.Although modeling efficiency can be greatly improved,the whole process is timeconsuming and it is difficult to achieve real-time performance.With the rapid development of deep learning,deep learning-based relief modeling has become a hot topic in computer graphics.Image is easy to acquire,but it is quite challenging to regress a 2.5D relief from it due to the lack of depth information.So far,some works have proposed deep learningbased methods for the modeling of calligraphy relief and flower relief,but deep learningbased solutions for portrait relief modeling have not been studied,mainly due to the lack of supervised training data.This project studies the problem of deep-learning based portrait relief modeling from single image,where the 3D portrait behind usually has complex geometric structure and fine details.We explore the key issues such as data generation,deep neural network construction network training.To meet different application scenarios,we propose two different solutions in this project: portrait bas-relief modeling and portrait relief modeling with adjustable thickness.The overall research contents are as follows:(1)Portrait bas-relief modeling: The thickness of portrait bas-relief is usually limited.To provide strong 3Dness,we need to construct a height field from single image with reasonable structure and rich geometric details.First,we take the RGB photos of Celeb AHQ as references,and propose a semi-automatic method to construct portrait normal maps,followed by an optimization-based method to reconstruct bas-relief samples from normal maps.In this way,we construct a large-scale relief dataset for supervised learning.Then,we design four kinds of convolutional neural networks for photo-to-depth mapping:UNet,Res UNet,Ghost Unet and Dense UNet.Through qualitative and quantitative analysis,we select Res Unet as the final mapping network.Experimental results show that the proposed method is capable enough to handle RGB photos with various lighting environments,facial expressions,hairstyles and skin colors,and can quickly generate high-quality portrait bas-reliefs without the need of manual interactions.(2)Portrait relief modeling with adjustable thickness: In order to meet the requirement of portrait relief generation with adjustable thickness,we further propose a novel method for portrait relief modeling,which takes single image and style vector as inputs to the network.To overcome the limitation of previous dataset that contains only portrait bas-reliefs,we synthesize portrait high-relief samples by taking 3D portraits as references,and train a neural network to infer portrait high reliefs from normal maps.In this way,we upgrade the relief dataset to contain not only bas-relief but also high-relief for each reference photo.Finally,we propose a Transformer-based network for photo-todepth mapping,which takes a single image and a style vector as inputs.Experimental results have proved the flexibility of the style-guided solution,which can not only generate portrait bas-relief,but also other type of portrait relief with adjustable thickness.