Research On Novel View Synthesis Based On Deep Learning

Novel-View Synthesis(NVS)aims to fully and realistically generate images from new viewpoints,and has long been an important challenge in the fields of computer graphics and computer vision.Applications of NVS technology include virtual reality and augmented reality,such as providing a more realistic experience in games,or for three-dimensional visualization of products in industrial design.In addition,it can also be used to reconstruct lost cultural heritage or monuments,or in areas such as scene prediction for self-driving cars.There are many methods for new viewpoint synthesis.This paper mainly studies the new viewpoint synthesis algorithm based on images.It refers to pre-shooting a set of scene images,using these pictures that have been taken,and synthesizing new scene images at unphotographed new viewpoints,so as to realize arbitrary 3D viewing effect of any viewing angle.In the traditional new view synthesis algorithm,3D image warping(3DWarping)is one of the most common methods.Its basic idea is:use the depth map to map the 2D texture to the surface of the 3D object.Next,the Points in 3D space are projected onto the image plane of the virtual camera.The key to the 3D-Warping technology lies in the accuracy of the depth map/disparity map.This paper uses the depth map/disparity map hole completion technology to improve the synthesis effect of the traditional new viewpoint synthesis algorithm.The advent of the machine learning era has opened up new ideas for new viewpoint synthesis.Combining the traditional 3D-Warping method with the neural network,a network DPNet based on the pixel interpolation algorithm(Image-Based Rendering Algorithm,IBR)is proposed.Use the pixel values of the source view pixels to calculate the pixel values of the destination view pixels.By improving the structure of the neural network and enriching the loss function of the network training,the hole problem of the depth map in the traditional 3D-Warping is avoided.Optical flow refers to the displacement of pixels on the surface of the same object in two image frames,usually represented by a two-dimensional vector.Optical flow can be used to describe the motion of pixels in an image.Inspired by optical flow,this paper proposes a new concept of feature flow.Feature flow is also a two-dimensional vector used to describe the relationship between the deep features of two images.sports relationship.With the help of feature flow,this paper proposes a pixel generation algorithm(Pixel Generation Algorithm,PG)based network FNet,which extracts features from the source view and finds the relationship between the source view and the target view to indirectly synthesize the target view.Depending on the applicable situation,when the pose transformation is small,the pixel interpolation IBR algorithm can be used;when the pose transformation is large,the pixel generation PG algorithm can be used.When the pose transformation is small,the source view and the target view have a large overlap.This feature can be fully utilized to directly interpolate the pixels of the source view to the target view;in this way,the target view can be restored.Therefore,the key to the IBR algorithm is to correctly estimate the depth map.When the pose transformation is large,the overlap between the source view and the target view is less,so the key to the PG algorithm is the extraction of source view image features.