Deep Learning Based Void Filling In Virtual Viewpoint Images

With the development of multimedia and computer technology,three-dimensional technology has a wide range of applications in virtual reality,three-dimensional video,and other fields.Its realistic visual appearance can bring users a different experience from two-dimensional video.In3 D technology,virtual viewpoint rendering technology can build multi view stereoscopic video.Among them,Depth Image Based Rendering(DIBR)can quickly draw other viewpoint images using reference viewpoint and camera position information.However,due to quality defects in depth images,rounding errors in the calculation process,and foreground occlusion,DIBR can cause cracks,artifacts,and voids in the rendered virtual viewpoint images,These rendering problems can seriously reduce the quality of rendered images and affect people’s viewing experience.The research direction of this article is to improve the quality of rendered images by solving the quality defects of virtual viewpoint rendering technology.The main research achievements and contributions are as follows:(1)This article conducts in-depth research and elaboration on DIBR technology.Aiming at the problems of cracks and artifacts in DIBR technology,a preprocessing method based on depth images is proposed.This method preprocesses the input depth image,including foreground edge detection and diffusion,virtual view depth value comparison,and noise removal operations.Pre processing can effectively deal with cracks and artifacts in DIBR images.(2)Aiming at the hole problem in DIBR technology,a network model based on context feature fusion was proposed to fill the hole area in the rendered image.It uses a homemade mask image and generates a confrontation network to generate structural information of the hole region;Then,an attention model combined with context feature fusion is used to improve the image quality of empty regions.This method can effectively deal with the phenomenon that the filling algorithm will incorrectly match the foreground information when the foreground objects in the virtual viewpoint sequence have fast motion.Experimental results show that this method is superior to existing methods for filling holes in rendered images.(3)Aiming at existing methods that only consider spatial information when filling holes in rendered images,a network model based on spatiotemporal feature fusion is proposed to repair the hole regions in rendered images.The model takes video as a unit and uses 2D(Two Dimension)convolution to convert a single frame sequence into low resolution features;Then,a spatiotemporal feature extraction module is used to extract temporal and spatial features from the input data.The context information in spatiotemporal domain features is fused in the network using a feature decoupling module,and empty spatiotemporal domain information is extracted from multiple dimensions through an attention matching mechanism.Experiments show that this method is superior to other video repair methods.