Research On RGBD Image Enhancement Based On Deep Learning

Depth is one of the indispensable information in many application scenarios,such as augmented reality,3D reconstruction,and unmanned driving.The RGBD camera can not only obtain depth map of the scene,but also obtain the color image/RGB image in the same scene.However,due to the limitation of the acquisition device,the resolution of the obtained depth map is much lower than that of the corresponding RGB image.It is difficult to meet the needs of practical applications.Aiming at this problem,the super-resolution reconstruction methods of depth map are studied under the condition that high resolution RGB image is used as the guide image,and the depth map is reconstructed by making full use of detail information of RGB image,so as to obtain high resolution and quality depth map.The main research work is as follows:(1)The method of the depth map super-resolution reconstruction based on an improved U-Net network is proposed.In order to utilize high-frequency features of input depth map and hierarchical features of the corresponding guidance image,UDepth SR for depth map super-resolution reconstruction based on U-Net is proposed.In this network,high resolution(HR)depth map is reconstructed by using the low resolution(LR)depth map and the HR RGB image through residual learning.UDepth SR adds the multi-level guidance branch(MGB)to U-Net network,which is used to extract multi-level intensity features from the color image to guide the reconstruction of the residual map.And it introduces the channel attention mechanism module(CAB)in the skip connection,which is used to highlight more useful channel information and extract high-frequency features of input depth map,thereby supplementing the Decoder’s upsampling process with finer features.The features extracted by the Decoder,hierarchical features extracted by the MGB,and highfrequency features extracted by the CAB are fused,getting the HR depth map through residual learning.The experimental results on public datasets show that the results of UDepth SR outperform some traditional methods and existing methods,thus verifying the effectiveness of the proposed network.(2)The method of depth map super-resolution reconstruction based on the multiscale residual block is proposed.In order to utilize multi-scale features of the depth map and high-frequency features of the corresponding guidance image,the network of based on multi-scale residual block(MSRNet)for depth map super-resolution reconstruction is proposed,which uses high-frequency features extracted from the RGB image to gradually guide the super-resolution reconstruction process.The network contains two branches: the high-frequency guided branch(HFGB)and the multi-scale reconstruction branch(MSRB).The high frequency layer(HFL)in HFGB uses Octave convolution to decompose the RGB image features into high-frequency features and low-frequency features,and only high-frequency features are used as guidance to provide useful high frequency details.Low-frequency features are suppressed to reduce parameters and maintain the reconstruction effect while improving the efficiency.MSRB includes the multi-scale residual block(MSRM),which uses convolution kernels of different sizes to extract depth features at different scales based on the residual structure,and uses skip connections between features at different scales to achieve feature sharing and reuse,utilizing deep features fully in the reconstruction process.high-frequency features extracted by HFL and multi-scale features extracted by MSRM are fused to reconstruct the residual map,and finally HR depth map is generated through residual learning.Experimental results show that the results of MSRNet are better than some traditional methods and methods based on deep learning,and this work has certain practical application value.