A Research On Depth Restoration And Recovering Based On Spatio-temporal Consistency

As the fast development of information technology and digital televisionindustry, China will soon embrace the digital TV era by stopping broadcastingthe anolog signal nationwide. The3DTV era, which is considered as the nextgeneration of TV system, has been deemed to come to reality as digital TV,which is only a matter of time.3D TV program breaks the concept oftraditional2D TV by transmitting two views simultaneously. By the terminalpart, users can get a more realistic and strong live feeling, by wearing3Dglasses to facilitate the fusion of the two views in their head, which greatlyincreases peoples’s visual impression. Recently, due to the success of3Dmovie such as Avatar and Titanic, there appears a tremendous interest in3Dmarket, with the wide spread of3D TV concept. Since Depth image containsample geometric information, it can easily find application in computer vision,such as Video Editing, Depth Image Based Rendering and Object Recognition.Thus, to succeed in these tasks, depth image is crucial, and the quality ofdepth image will directly decide result of these applications.In this paper, we focus on the processing to depth images, includingactive acquired depth images and passive acquired ones. For the activeacquired depth image, we investigated the Microsoft Kinet device andanalized its depth images. To restore the missing pixels in Kinect depthimages, we have proposed an algorithm. Our method refer the missing pixlesby using technique of plane fitting and Region Consistency Metric based jointbilateral filters. The reultls show that our method can restore the missinginformation in Kinect depth image faithfully. For passive acquired depthimage, we present a method for dense depth estimation taking the input of atrinocular video. The algorithm works with a global energy minimization framework based on Markov Random Field (MRF). The visibility/occlusionconstraint and spatial-temporal consistency are explicitly considered in aniterative fashion. Depth maps are initialized using AD-Census metric andBelief Propagation, followed by Mean Shift Segments Fusion. We furtherincorporate the visibility and spatial consistency constraint to refine our depthmap. In order to make the inference tractable, we implement our algorithm inan iterative scheme where the disparity of each view and occlusion maps areupdated iteratively. In terms of the temporal consistency, we adopt OpticalFlow to construct the correspondences of pixels among neighboring frames,and modeled the temporal coherence in our energy function for optimization.Finally, a quadratic polynomial interpolated sub-pixel recovering process isperformed to suppress the quantization artifacts of the depth map. Our methodaddresses the problems such as untextured regions, occlusions, inconsistencyin spatial-temporal domain in a unified framework. The experimental resultsshow that our disparity maps are accurate and highly consistent inspatial-temporal domain without oversmoothing the boundaries.