Learning-based Depth Map Up-sampling Algorithm

The rapid development of information technology has made image and video important information carrier.Depth image plays an extremely important role in the visual tasks such as object detection and 3D reconstruction.However,obtaining accurate depth images has always been a major challenge in the field of computer vision.At present,depth cameras,such as TOF and Kinect,are widely used for acquiring depth images due to their real-time and high-precision.Besides,under the limitations of imaging conditions and the interference from the external environment,the depth images captured by these devices suffer from low-resolution and are quite susceptible to noise.Presently,the theory of sparse representation and deep learning,as the typical learning-based image sampling method,has become a hotspot in the field of image processing.Learning-based image up-sampling uses a large amount of training data to obtain useful features,and building a non-linear relationship between low-resolution input and high-resolution output by constructing dictionaries or network models to achieve the reconstruction of the low-resolution image.Not only can this kind of algorithm achieve the resolution of image improvement,but also preserves more details of the image.As one of the most successful up-sampling algorithms,the learning-based upsampling method is also faced with the problems of inaccurate edge information and noise interference,while applied to depth maps.In addition,the virtual viewpoints obtained by these reconstructed depth maps are prone to artifacts and j aggies,resulting in unsatisfactory effects.In this paper,the depth map upsampling technique based on learning is studied in detail.The main research achievements are as follows:(1)Based on the theory of sparse representation,an improved algorithm of depth upsampling with texture edge feature via sparse representation is proposed.With the guidance of the texture edge feature,the gradient features of depth image are extracted to establish the relationship between low resolution depth map and high resolution output,and the guided filtering is applied to the high resolution depth images of training set and test images.While ensuring the accuracy of the edge information,the noise can be eliminated to a certain extent.Both the upsampled depth map and the synthesized view can be improved effectively.(2)Using the convolutional neural network as a model,this paper proposes the depth map upsampling method using joint edge-guided convolutional neural network for virtual view synthesizing.The proposed network combines the depth map with joint edge feature as two channels’ input,and then utilizes the edge texture to build the local enhancement constraints,which are effective for preserving the edge regions of the depth map.Besides,the global optimization for the enhanced depth map provides depth image of high quality as the final output.With the guidance of the joint edge feature,the constraints of local enhancement conditions and the global optimization,the effectiveness of the proposed method has been verified by adequate experiments.(3)The inter-frame correlation brings about redundant information for depth video sequences.It will result in extra memory consumption and computational cost if the sequence is upsampled entirely.Therefore,the algorithm of depth video sequences upsampling is proposed.Motion vector based the key frame extraction is introduced to produce all key frames,and only the edge patches in the key frames are trained.In the original three-layer network,the feature enhancement layer is added to enhance the feature learning,which not only solves the problem of inaccurate upsampling in the edge region,but greatly reduces the computation complexity.