Dynamic Human Free View Synthesis Based On Multi-View Inputs

Human free view synthesis based on multi-views inputs has long been an important research issue in the field of computer vision and computer graphics.It aims to synthesize the arbitrary view image or video of the human from multi-view images or videos.At present,it has been applied to AR/VR,sports event broadcasting,games,film industry and other fields.Compared with the use of a dense array of cameras system or depth sensor for human view synthesis or 3D reconstruction,the task from sparse multi-view inputs has a wider application range and lower cost,but it is also more challenging.In recent years,with the rapid development of deep learning technology and the continuous improvement of the performance of computing devices,neural rendering related methods are gradually making up for the shortcomings of traditional methods.Among them,the method based on neural radiance fields(NeRF)has attracted wide attention due to its impressive performance.At present,human free view synthesis and reconstruction based on the NeRF method still faces many challenges,including the following main problems: free view synthesis of dynamic human,free synthesis of new human,and human reconstruction.This paper focuses on the above three issues,uses the parametric model SMPL as the prior information of the human to improve the existing NeRFbased methods,and verifies the feasibility of the proposed method with the data set produced by the multi-view camera acquisition system and the public dataset.The specific research contents are as follows:(1)To solve the problem of dynamic human free view synthesis lack of details,this paper proposes a dynamic human neural radiance field.The model utilizes SMPL texture map and vertices to provide prior information about the appearance of the human.First of all,the texture map of SMPL model is used to build a texture feature map that can be shared by different human poses.Then,the latent codes representing the appearance and geometry of the human are placed on the vertices of the SMPL model,which are diffused into the 3D space using the Sparse Conv Net.Finally,the human image is synthesized through the volume rendering technique of NeRF.The experimental results show that this method can achieve better visual view synthesis results in both trained and untrained human poses.(2)To solve the problem of poor generalization ability for new human free view synthesis,this paper proposes a generalizable neural radiance field guided by geometric model.On the basis of using SMPL model as a prior geometric model,the network adds sparse view image information and depth information of human as additional inputs,thereby synthesizing images of new human,that is,untrained human.First of all,a color blending module that can judge occlusion is designed with the help of depth information,which can better aggregate multi-view information to obtain the appearance of the human.Then,the geometric shape of the human is deduced using the point cloud and multi-view consistency features provided by the SMPL model.Finally,the human image is synthesized by volume rendering.Experimental results show that this method can not only render images of new target human,but also effectively handle the self-occlusion problems in the case of sparse view inputs.Compared with other state-of-art methods,this method has stronger generalization ability.(3)To solve the problem of poor geometric details in human reconstruction,this paper proposes a human reconstruction model based on surface field and radiance field.In this model,the signed distance function of the surface field replaces the volume density expression of the radiance field.First of all,the sampling points in the observation space are mapped to the canonical space using the human motion.Then,the information of multi-view image fusion is used to calculate the offset of the sample points transformed into the standard space from the surface.Finally,the neural network is used to store the geometric information and color information of the canonical space human model,and the human mesh is reconstructed with the help of volume rendering technology and Marching Cubes algorithm.The experimental results show that the method is better than the reconstruction method based on the volume density of radiance field.