Multi-Stream Convolutional Neural Network For Depth Estimation From Light Field Images

Light field(LF)imaging has attracted much attention because of its unique imaging principle.The images taken by traditional cameras record the position information of the light projected onto the two-dimensional(2D)plane,exclusively,ignoring the direction information of the light propagation.The light field collected by the dense micro-lens array equipment can record the four-dimensional(4D)information,i.e.the position information and direction information,of the light propagation.According to the 4D information of the light field,there are three forms for the light field visualizations:lenslet image,sub-aperture image,epipolar plane image(EPI).Utilizing the visualized light field image,the depth of the image can be reconstructed.There are four light field image depth estimation algorithms:multi-view-based algorithm,epipolar-plane-image-based algorithm,refocus-based algorithm,and deep-learning-based algorithm.As the diversity of the real scenes,the performance of each depth estimation algorithm is limited to varying degrees.The light field image areas with simple texture produce the errors of depth estimation,frequently,and the depth estimation result is affected by noise.Combining the theories and advantages of four algorithms,this paper proposes a light field image depth estimation algorithm based on multi-stream convolutional neural network.Firstly,we preprocess the light field image.The single image super resolution(SISR)algorithm is applied to perform spatial super resolution of the sub-aperture image,which can increase the number of the image pixels.We extract the sub-aperture images on horizontal(row),vertical(column),45-degree and 135-degree,which include the center sub-aperture image,as four-flow sub-aperture images input.Secondly,we classify the texture of the light field image.We calculate the gradient value of the pixels in the image,and compare it with the given gradient threshold,which can divide the image area into two categories:simple texture and complex texture,and label them.Thirdly,we extract the EPIs of the light field image in four directions:horizontal(row),vertical(column),45-degree and 135-degree,which applied to generate four-stream EPIs.Next,we perform multi-scale observations on EPIs.According to the labels of texture classification,put the different texture area blocks into different scales convolution to extract features.Blocks of simple texture label use large convolution kernels and the simple ones use small convolution kernels for extracting features.Finally,we apply the fusion network to cascade the features of the four-stream EPIs,which can form higher-dimensional features,so as to find the correlation between the features by matching,and obtain the depth estimation results.According to calculating the average absolute error(MAE)of the light field depth estimation map and its ground truth,we analyze the performance of our algorithm,quantitatively.The experimental results showed that,when testing the noise-free light field images,the average MAE between the depth estimation of proposed algorithm and ground truth is reduced,compared with other depth estimation algorithm,which can verify the accuracy of our proposed algorithm.When testing the noise-contaminated light field images,compared with other depth estimation algorithms,the average MAE is still reduced.And difference between the two depth estimation MAE is also reduced,which can verify the robustness of our proposed algorithm.In subjective evaluation,the depth maps obtained by the algorithm proposed in this paper perform well in the local area with simple texture,so that the algorithm in this paper can avoid the errors of depth estimation in simple texture areas,effectively.