Research On Multi-view Depth Acquisition And Quality Assessment

With the rapid advancement of video technology,3D video (3DV) and free viewpoint video (FVV) have attracted great attentions and extensive research. Compared with traditional 2D video,3DV can bring more realistic experience of the scene, and FVV enables interaction between the audiences and the content providers. In such novel systems, multiview depth maps are used to generate virtual views, which not only reduces the data volume, but also makes the system flexible. Therefore, multiview depth is a fundamental and also a key factor. Depth maps indicate the distance between the scenes and the imagining planes, and thus are only attainable through measurement or calculations. The approach to acquiring high quality depth maps and the method of depth quality assessment are of significant importance in 3DV and FVV systems. In this dissertation, we will study this topic in three aspects. First, multiview depth acquisition with RGB-D cameras is studied, where the inter-device interference is the main concern. Afterward, considering that the off-the-shelf depth cameras cannot meet the demand of novel video applications, we propose a novel hybrid depth acquisition scheme after reviewing the existing structured light based depth acquisition algorithms. Finally, due to the lack of error-free reference, depth maps can only be assessed in a no-reference fashion. To meet this demand, we propose a scheme based on the intrinsic physical meaning of depth maps to detect their errors and to quantify their quality.In the past decade, depth cameras have obtained great success, especially RGB-D cameras, i.e., Kinect developed by Microsoft, have facilitated the depth acquisition in an accurate and convenient mannar. However, the lack of cooperation mechanism among RGB-D cameras causes inter-device interference when multiview depth maps are captured, which degrades depth quality greatly. In this dissertation, we study the cause and the negative effects of the interference problem, and therefore propose a scheme to cancel the interference and recover the depth maps. Statistical results show that, interference causes depth missing rather than errors, and it is sufficient to recover the missing depth values from the available ones. On the other hand, considering that depth maps have quite different properties inside objects and across boundaries, we make the scheme region- adaptive. To be specific, the inner regions and boundary regions are first classified with the help of texture maps. For the smooth inner regions, a Markov random field (MRF) model is used in the gradient domain, followed by the use of discrete Poission equation (DPE) to calculate depth values. In contrast, for the boundary regions, another texture-guided MRF model is utilized to compute depth values directly. This scheme can keep the inner regions smooth while preserve sharp object boundaries.In addition to the interference problem, the off-the-shelf RGB-D cameras cannot meet the demand of new video applications. High quality depth maps should be accurate, dense, attainable with low-complexity, and can be extended to the scenario of multiview depth acquisition. Therefore, in this dissertation, we propose a novel high quality depth measuring scheme based on the existing structured light techniques. In this scheme, a novel band-based sinusoidal pattern is designed for projection. This pattern is sinusoidal in every band, which enables it to carry phases, and the pattern is also locally unique, which enables a hybrid approach for depth measurement. To be specific, within each band, the fringe is sinusoidal and the wrapped phases can be obtained through Fourier transform profilometry (FTP). After that, a novel phase unwrapping algorithm is performed based on the local smoothness property of the depth maps and the primier periods of the decoding bands. Afterward, the unwrapped phases are converted to depth values. On the other hand, for those regions where the local smoothness assumption is not satisfied, active stereo matching is applied to further revise the depth values. Experimental results demonstrate that the proposed scheme can generate high quality depth maps even in complex scenes with huge depth abruptions and spatially isolated objects. In addition, combined with propoer multiplexing techniques, the proposed depth measuring scheme can generate multiview depth maps.Last but not least, depth maps are obtained through measurement and calculations, so depth errors are inevitable and should be detected. On the other hand, error-free reference depth maps are not available, which makes common full-reference and reduced-reference image quality assessment inapplicable. In this dissertation, we propose a no-reference scheme to solve the problem. This scheme is based on the intrinsic physical meaning of depth maps, where the geometry distortion of depth edges is the focus of our work. With the absence of reference depth, we achieve accurate matching between the depth edges and the texture edges because the corresponding depth and texture maps are highly related. To be specific, the texture and the depth maps are two forms of the same scenes, and thus their edges are similar. In the proposed solution, three edge features:spatial location, edge orientation and edge length, are used to match depth and texture edges accurately. In addition, segment-based matching, instead of pixel-based matching, is adopted to improve the matching robustness. Finally, with the matched edge pairs, the distorted depth edges can be obtained, together with a quantitative depth quality index being computed. Experimental results demonstrate that the proposed scheme can report bad pixe!s along depth edges accurately and can indicate boundary distortion. The proposed no-reference quantitative metric not only coincides with the full-reference metric well, but also is highly related to the quality of synthesized virtual views.At the end of this dissertation, we summarize our work and contributions in the research topic, and discuss the possible future work.In general, this dissertation explores the issue of multiview depth acquisition and no-reference depth quality assessment, and it can innovate the research and application in depth-based systems such as 3DV and FVV.