Research On Depth Map Optimization And Coding For3D Video

3D video is an emerging new media for rendering dynamic real-world scenes. Comparedwith traditional2D video,3D video is the natural extension in the spatial-temporal domainas it provides the depth impression of the observed scenery. Besides the3D sensation,3Dvideo also allows for an interactive selection of viewpoint and view direction within thecaptured range. An attractive3D video representation is multi-view video plus depth(MVD) format. With the help of depth maps, many interesting applications such asglasses-free3D video, free-viewpoint television (FTV), and gesture/motion based humancomputer interaction are becoming possible. However, MVD results in a vast amount ofdata to be stored or transmitted, efficient compression techniques for MVD are vital forachieving high3D visual experience with constrained bandwidth. Consequently, efficientdepth coding is one of the key issues in3D video systems.Depth maps are used to synthesize the virtual view at the receiver side, so accurate depthmaps should be estimated in an efficient manner for ensuring a seamless view synthesis.Although the depth cameras provide depth data conveniently, depth maps from thestructured light cameras contains holes owing to their inherent problems. In this paper, wepropose a hybrid multi-scale hole filling method to combine the modeling strength of theparametric filter and nonparametric filter. We progressively recover the missing areas inthe scale space from coarse to fine so that the sharp edges and structure information in thefinest scale can be eventually recovered. For inter scale, this paper presents a novel linearautoregressive-based depth up-sampling algorithm considering the edge similarity betweendepth maps and their corresponding texture images as well as the structural similarityamong depth maps. For intra scale, this paper propose a weighted kernel filter for holefilling based on a weighted cost function determined by the joint multilateral kernel. Thismethod can remove artifacts, smoothing depth maps in homogeneous regions and improving the accuracy near object boundaries.A key observation is that depth map is encoded but not displayed; it is only used tosynthesize intermediate views. The distortion in the depth map will affect indirectly thesynthesized view quality, so the depth map coding aims to reduce a depth bit rate as muchas possible while ensuring the quality of the synthesized view. In this paper, we propose adepth map coding method based on a new distortion measurement by deriving therelationships between distortions in coded depth map and synthesized view. We firstanalyze the relationship between depth map distortion and geometry error by mathematicalderivation, and then set up a model to describe the relationship between geometry error andsynthesized view distortion. Based on the two mathematics relationships, the synthesizedview distortion due to depth map coding is estimated.More specifically, depth coding-induced distortion in synthesized view is always alongboundaries, which is significant for human eyes. However, mean squared error(MSE) andthe like that have been used as quality metrics are poorly correlated with human perception.In this paper, we apply the structural similarity information as the quality metric in depthmap coding. We develop a structural similarity-based synthesized view distortion (SS-SVD)model to capture the effect of depth map distortion on the final quality of the synthesizedviews. The model is applied to the rate distortion optimization which describes therelationship between depth coding bit-rate and synthesized view distortion for depth mapcoding mode selection. Experimental results show that the proposed SS-SVD methodobtains both better rate distortion performance and perceptual quality of synthesized viewsthan JM reference software.Depth maps generally have more spatial redundancy than natural images. This propertycan be exploited to compress a down-sampled depth map at the encoder. In this paper, wepresent an efficient down/up-sampling method to compress the depth map efficiently. Anovel edge-preserving depth up-sampling method is proposed by using both the texture anddepth information. We take into account the edge similarity between depth maps and their corresponding texture images as well as the structural similarity among depth maps tobuild a weight model. Based on the weight model, the optimal minimum mean square error(MMSE) up-sampling coefficients are estimated from the local covariance coefficients ofthe down-sampled depth map. The up-sampling filter is combined with HEVC to increasecoding efficiency. Objective results and subjective evaluation show our proposed methodachieves better quality in synthesized views than existing methods do.