Multi-view Stereo Based On Spatial Pyramid Pooling For Light Field Depth Estimation

Light fields describe our world with richer information and have raised a lot of attention as a scene representation recently.Traditional photography simply records a two-dimensional projection of the light upon the focal plane,and the angular domain which describes structure of the scene has been hidden in the two-dimensional projection representation of an integrated way;while light fields collect radiance from rays in all directions,record information in the angular domain explicitly,and demultiplex the structure information of the scene.Depth estimation from light field image is a crucial basis for light field relative applications.Since the multi-view representation of light field is contained with rich,abundant and available information,how to extract features from these views and fuse the features in an effective way become a key point for accurate light field depth estimation.In the article,we study into light fields and we focus on how to obtain accurate light field depth estimation results.The main research contents are as follows:(1)To address the problem of how to apply the light field theory for practical usage and how to construct the light field data structure effectively,combined with the actual engineering application,we simplify and resample the light field representation of the plenoptic function and give three data structures for two-dimensional visual representation of four-dimensional light field: multi-view representation based on subaperture image,representation based on sub-view of light field and representation based on epi-polar plane image.(2)For the optimization-based method of the light field depth estimation,we have developed a "first initial depth estimation,then depth optimization" approach.In the initial depth estimation,we study into the matching method based on sub-aperture image and the method based on epi-polar plane image;in the depth optimization,we introduce the Markov random field framework process and the variational model;a quantitative comparison of the estimation results of each method and an experimental comparison are also presented.(3)For a deep learning approach to light field depth estimation,we propose a multi-view correspondence network based on spatial pyramid pooling.Based on the multi-view representation of the light field sub-aperture image,we obtain features with contextual information by applying the spatial pyramid pooling module to all the views in the light field,maximizing the use of the rich and redundant information in the light field.As to the correspondence part,through stereo correspondence intuition,we first build a cost volume,then apply cost aggregation in the light field though 3DCNN,and finally obtain the final depth estimation map by regression.We give the quantitative results of our method and a comparison of other methods.The experimental results show that our method outperforms the other methods in detailed representation and has fewer regions of depth uncertainty.