Research On Depth Generation And Upsampling Methods For Three Dimensional Video

With the development of digital multimedia processing technology and the increasing demand for the most realistic visual experience,three dimensional television(3DTV)has rapidly become a new research and application hotspot after the successful application of high-definition digital television.Based on a novel data format including two-dimensional color video and its corresponding depth information,modern 3DTV employs the depth based image rendering(DIBR)technology to generate the virtual views,and then its stereoscopic display technology provide users the three-dimensional immersive visual experience.The unique data format,i.e.two-dimensional color video plus its corresponding depth video,is an important basis for 3DTV system,obviously,and depth maps play a very important role in this system.Thus,acquiring high-resolution(HR)depth maps is the focus of research in the business world and academia.Generally,the Time of Flight(TOF)depth measurement method can simultaneously capture the color video and depth data of the scene,but there are physical differences in resolution between those two videos.It is necessary to increase the depth video resolution to satisfy the practical applications.In addition,due to the existing rich 2D video resources and lack of 3D video resources,2D to 3D video information conversion can effectively solve the dilemma.Most 2D-to-3D conversion algorithms need to generate depth maps,however,and the manual direct depth extraction methods require a lot of manpower and time costs.In response to the above problems in the acquisition of HR depth maps in 3DTV,this paper focuses on the depth up-sampling and depth generation techniques,and major contributions of this dissertation are summarized as follows:1.Depth upsampling via pixel-calssifying and joint bilateral filteringA depth image up-sampling method is put forward by using pixel classifying and jointed bilateral filtering in this paper.By analyzing the edge maps originated from the high-resolution color image and low-resolution depth map respectively,the pixels in up-sampled depth maps can be classified into four categories:edge points,edge-neighbor points,texture points and smooth points.Firstly,the joint bilateral up-sampling(JBU)method is used to generate an initial up-sampling depth image.Then,for each pixel category,different refinement methods areemployed to modify the initial up-sampling depth image.Experimental results show that the proposed algorithm can reduce the blurring artifact with lower bad pixel rate(BPR).2.Edge-guided with gradient-assisted depth up-samplingMost of depth up-sampling algorithms are based on the consistent hypothesis,i.e.,the object boundaries or texture regions in the color image are consistent with depth discontinuity regions in the depth map.However,the hypothesis is not always correct.Under the combined guidance of HR depth edge map and HR color image gradient map,this paper presents a simple and efficient depth up-sampler.Firstly,the consistent regions are distinguished from the other regions and more accurate depth edge points are founded.Then,the up-sampled depth map from traditional bilinear interpolation are refined by using effective depth-assignment schemes.Extensive experiments demonstrate that the proposed method outperforms conventional interpolation algorithms and some other edge-based depth up-sampled methods,and it can effectively improve the up-sampled depth map quality with low computational complexity.3.Geometry complexity approach for image segmentation and depth generationWe present an image segmentation and depth generation algorithm based on geometry complexity,which is applied to outdoor scenes,in the paper.Firstly,the angle statistical distribution of main lines in the input image is calculated and then the outdoor scenes are classified into four geometric categories.Secondly,the input image is divided into many small regions using the mean-shift segmentation algorithm,and then these regions are merged into three big regions based on the scene geometry category results.Each big region is in coherent depth distribution.Thirdly,a depth map can be generated based on the geometry segmented result and its standard depth map.Experimental results show that proposed method can obtain an effective image geometric segmentation and meanwhile get a depth map with better details and more close to the true depth information of the scene.4.Single outdoor image depth map generation based on scene classification and obj ect detectionWe present an efficient depth generation framework from a single image,which is based on scene classification and object detection.In view of the fact that similar 3D scenes usually have similar depth maps,we firstly construct multi-scene image dataset with different scene structures,and each scene type contains many color images and corresponding depth maps.Secondly,the k-nearest neighbor(KNN)algorithm is employed to judge the scene category of the input image in the image dataset,and then the initial depth map is obtained by fusing the depth maps in the category.Finally,we incorporate the image segmentation results to detect the sky region and the ground region by using color information,and the depth map is obtained by improving the initial depth map based on the depth characteristics of the sky and ground region.Experimental results demonstrate that the proposed scheme can generate smooth and reliable depth maps with satisfied performance.