Depth Map Super-Resolution Based On Information Fusion With Multiple Constraints

The depth map captured from low-cost depth camera such as Kinect frequently suffers from low-resolution,high noise and data missing.Therefore,how to remove the noise and meanwhile increase the resolution of depth map becomes an exigent and challenging task.In this paper,several issues related to depth map super-resolution(SR)are deeply researched under an unified framework by fusing multiple constraint information and some progresses have been made.The major contributions of this thesis are as follows.First,a pre-processing algorithm is proposed to fill the holes in raw depth map.Inspired from the image inpainting method based on the fast marching method(FMM),the probability contour information from color image is combined into the FMM to de?termine the inpainting order of pixels in hole-regions,and make the pixel that is nearer depth edge and closer to hole boundary has a higher inpainting priority.Furthermore,in order to eliminate the artificial blurring in some repaired regions,the inpainting depth values are refined by combinedly utilizing bilateral filtering and non-local means filter-ing under the guidance of color image.The experiment results show that our algorithm obtains good performance in depth hole-filling and provides better initial depth data for the following depth SR work.In order to obtain smoothness and reality high-resolution depth map,a depth SR approach is proposed by combing the raw depth map and the non-local total general-ized variation(NLTGV)in a convex variation model.In virtue of multiple constraint information such as color and multi-level segmentation,the model possesses ability to generate sharpness depth edges that is related to local structure of scene.Further-more,the model is modified by considering the histogram information of raw depth around the neighboring pixels and the texture energy of color image,and then our algo-rithm obtains more accurate depth SR result and better performance in suppressing the texture-transfer.The depth SR model with unified and fixed tuning parameters is difficult to gain a good balance between the performances of edge-preserving and suppressing the texture-transfer.To overcome this shortage,a novel adaptive depth SR model based on the NLTGV regularization is proposed in this paper.A fuzzy logic system is constructed to inference out the values of adaptive parameters for each pixels by considering the local structure features of depth map and the texture distributio of color image.Evaluations demonstrate that our algorithm obtains better performances in the smoothness,edge-preserving and suppressing texture-transfer than various typical algorithms.In order to further improve the quality of depth SR,the pixels of depth map are explicitly divided into four classes:pixels near the depth edge,non-edge pixels with rich color-textures,non-edge pixels without salient textures,and other pixels.Then,the weight parameters of SR model are classifiedly assigned according to the category of pixels.The labels of pixels are determined with a random forest based classifier that is trained by using several features with relatively stronger classification ability,such as magnitude of gradient and texture energy.Evaluation results demonstrate that the classified weights scheme can contribute to improve the accuracy of depth map.In summary,for an image-guided depth SR scheme with performances of strong adaptability and high accuracy,in this thesis several studies revolving around the sub-jects of data pre-processing,depth SR model construction and enhancing the adaptabil-ity of the model have been made by fusing multi constraint information such as the raw depth data,NLTGV smoothing prior,color image,multi-level segmentation,and together utilizing various features extracted from the input images.