Research On 3D Perception Based Virtual View Rendering

With the constant pursuit of video viewing experience,various of new video applications appear in recent years,which has led to the development of computer network and multimedia technology.Free viewpoint video(FVV)allows people to choose viewpoint freely,brings them an unprecedented immersive interactive viewing experience.However,the amount of data required by the system is quite large,and higher requirements are imposed on various processes such as capturing,encoding,transmission,decoding,rendering,and display.Therefore,the FVV adopts the depth map based rendering method(DIBR),which uses a limited number of reference viewpoints to draw arbitrary virtual viewpoints.It greatly reduces the data captured and transmitted by the FVV.The core of DIBR is using the depth information to reconstruct 3D space of the scene,and further realize the viewpoint mapping.In order to improve the virtual view visual quality,this paper studies the 3D perception based stereo virtual view quality assessment and the virtual view hole filling algorithm based on depth background modeling.(1)The stereo virtual view mainly includes two types of distortion: single view rendering distortion and stereo view mismatch distortion.For single view rendering distortion,the difference region between the distorted image and the reference image of current view is extracted firstly.Then the mean structural similarity index(SSIM)is calculated for this region,and finally get the monocular texture feature by averaging the left and right view indexes.For stereo view mismatch distortion,the left and right view distorted images are mapped to each other firstly.And the different region between the mapped and the reference image is extracted as binocular mismatch region.Then the SSIM index will be calculated for this region.Finally,the left and right views are averaged as the binocular competition feature.Lastly,the two features are converged as the objective quality index of stereo virtual view.Experimental results show that our method matches the DMOS effectively,the Pearson linear correlation coefficient and the spearman rank order correlation coefficient are 0.911 and 0.900 respectively,the stereo virtual view quality is correctly reflected.(2)Hole filling is the key issue in free viewpoint video rendering process.Most of the holes are located in visually significant areas,and incorrectly filling will greatly reduce the subjective quality.This paper proposes a new hole filling algorithm using depth background modeling.Firstly,the causes of the holes are analyzed,and the holes are divided into three types: the disocclusion regions,the structural holes and the cracks.Then,based on the depth pixel time domain histogram statistical features,the moving foreground and the time domain stable background are separated,the depth background is obtained,and the texture background is constructed accordingly.Then,combining the depth information,the warped texture background and the warped current frame texture image are merged to fill the disocclusion regions,which significantly affects the subjective perception.Finally,a depth-based image restoration method is proposed to fill the remaining structural holes and the cracks.The experimental results show that the proposed method effectively fills the holes in the virtual view,and is superior to the core comparison literatures in terms of rendering quality and time domain continuity.