Anti-specular Light-field Depth Estimation Using Convolutional Neural Networks

The light field camera,a new type of multi-view imaging equipment,can obtain multi-view and refocused images by recording the spatial and angular information of the scene in one shot.It provides rich scene geometrical information,which has a unique advantage in depth estimation.Specular is one of the most difficult problems in light field depth estimation.The existing methods usually assume that the objects in the scene are on a Lambertian surface or a uniform reflection coefficient surface.However,these methods often cannot predict accurate depth maps in the scenes with specular or non-Lambertian surfaces.Therefore,we explore the characteristics of image specular phenomenon for depth estimation in specular scenes.Most current studies ignore the influence of the context information on the specular surfaces in the scene.Inspired by the relationship between the multi-view,refocusing characteristics of the light field image and specular phenomenon,we focus on the algorithm research and discussion of the specular problem in light field depth estimation.The main work of the thesis is listed as follows:(1)We describe the background and research development of light field depth estimation,image specular and deep convolutional neural networks,and analyze the significance of the research topic.In addition,we introduce related knowledge theories such as light field definition,parametric representation,image representation and depth information extraction.(2)For the detection and removal of specular in light field images,we define scene types with different reflection based on the Dichromatic Reflection Model.We exploit light field multi-view and refocusing characteristics to construct point and line consistency measurements.Then we detect diffuse pixels and specular pixels,separately,and finally use the image confidence principle to remove specular pixels.Therefore,we have implemented a light field specular detection and removal algorithm based on points and lines consistency,which effectively alleviated the damage of specular to the image texture and well preserved the color distribution of the original image.Through comparative experiments,we show that the specular removal can improve the performance of light field depth estimation.(3)The specular phenomenon behaves differently in the context information of different surfaces,different viewing angles,and different image ranges.Therefore,we design and implement an anti-specular light field depth estimation network based onthese characteristics.On the one hand,we construct a multi-view branch based on the principle that the specular area changes as the viewing angle,and then obtain the depth information of the specular area under different viewing angles.On the other hand,we use dilated convolution to expand the receptive field of the network and obtain a wider range of context information.The network can learn a wider range of multi-view information,and effectively alleviates the effect of specular for depth estimation.In addition,we design a multi-scale feature fusion method by concatenating dilated convolution features with multi-dilation-rates and multi-kernels convolutional features,which can further improve the accuracy and smoothness of the depth estimation.Experiments show that our network can effectively estimate depth information from light fields.In particular,the depth information of the specular area has high accuracy of restoration,and the edge area of the object is smooth,and better preserve the image detail information.