Light Field Super-Resolution Via Fusing Multiple Correlations

4D light field imaged have wide applications in biomedical science,industrial optical inspection and deep-water exploration.However,light field images always suffer from low spatial and angular resolutions,which hinders their applications in practice.Therefore,light field image super-resolution becomes an important research topic in computational imaging.Recently,light field image super-resolution(SR)techniques based on deep convolutional neural networks(CNNs)have seen great progress.However,existing methods still suffer from low reconstruction fidelity,large model parameter amounts and poor generalization ability.To solve these problems,this thesis focuses on three aspects including performance improvement,lightweight networks,and model generalization.For each problem,this thesis exploits and fuses multiple forms of available correlations within light field images,then designs algorithms accordingly.The main contributions are listed below:For performance improvement,this thesis proposes a light field image SR algorithm by jointly exploiting internal and external correlations.The proposed algorithm takes merits of both internal and external methods and improves the reconstruction fidelity significantly.In specific,this thesis first advances the classic projection-based method to make better use of the internal correlation and designs a CNN based on alignment and aggregation to make better use of the external correlation.After that,through quantitative and qualitative analysis on these two methods,this thesis finds that the internal and external correlations are complementary to each other.Accordingly,this thesis propose a pixel-wise adaptive fusion network to take advantage of both their merits by learning a weighting matrix.Experimental results on both synthetic and real-world light field datasets validate the superior performance of the algorithm over the baseline methods.For lightweight networks,this thesis proposes a light field image SR method by fusing spatial-angular self-correlation and cross-correlation.The proposed method uses simple 2D convolutions to build feature embedding modules,which exploit and combine the information from the spatial dimension and the angular dimension within 4D light fields.The proposed method not only builds a lightweight network,but also achieves superior reconstruction performance.Specifically,this thesis analyzes the drawbacks on geometry information modeling of mainstream lightweight module,i.e.,spatial-angular separate convolution(SAS-conv)and accordingly introduces spatialangular correlated convolution(SAC-conv)to take use of the spatial-angular crosscorrelation.After that,through detailed error and feature analysis,this thesis validates that SAS-conv and SAC-conv are complementary to each other and proposes a novel module,i.e.,spatial-angular versatile convolution(SAV-conv)to take merits of them both.Experimental results on both spatial and angular super-resolution validate that,with fewer parameters,networks based on SAV-conv notably outperform those with SAS-conv and achieve state-of-the-art performance.For model generalization,this thesis proposes a zero-shot learning framework for light field image SR by fusing the cross-scale and cross-view correlations.The proposed method forms the zero-shot learning framework to exploit the cross-scale correlations and builds CNNs to use the cross-view correlations.Then the method combines these two correlations with specified training strategies to take full use of the internal correlations within a single light field image.In specific,the framework learns a mapping to super-resolve the reference view with examples extracted solely from the input low-resolution light field image itself.Meanwhile,based on the observation of the performance different networks have with limited training data under the zero-shot setting,this thesis proposes the divide-and-conquer strategy for network design and training to exploit and combine the cross-view correlation for better reconstruction.Moreover,this thesis introduces the error-guided finetuning algorithm based on the zero-shot learning framework to further combine the external information brought by an external source dataset and improve the robustness as well as generalization ability.Comprehensive experimental results validate that our method not only outperforms classic non-learningbased methods,but also generalizes better to unseen light fields than state-of-the-art deep-learning-based methods when the domain gap is large.