Research On Super-resolution Reconstruction Algorithm Of Camera Array

Along with the development of an era,high-quality images are urgently needed in many practical applications.Image resolution is a key indicator of image quality evaluation,which marks the richness of the image describing the details of the scene.As existing imaging systems are limited by hardware limitations such as the density of their inherent sensor arrays,the image resolution sometimes fails to meet the requirements.The image super-resolution reconstruction technology provides a solution to improve the image resolution from the software aspect through mathematical modeling and algorithm,which can improve the image resolution without changing the hardware of the imaging device.It has a great practical significance.According to the number of images processed in super-resolution reconstruction,it is generally divided into single-image super-resolution reconstruction and multi-image super-resolution reconstruction.Limited by the details contained in the input image itself,single-image super-resolution reconstruction is often difficult to achieve the desired effect.Compared with the single-image super-resolution algorithm,the multi-image super-resolution algorithm has the advantage of making full use of the complementary information between the image sequences.Therefore,multi-image super-resolution algorithm tends to have a stronger super-resolution capability.However,it is often difficult to capture image sequences with complementary information of the same scene.The camera array system can capture images of the same scene from multiple perspectives,so that the problem of image sequences collection with complementary information can be well solved.Based on the camera array system built by ourselves,considering the special properties of the image sequences captured by the camera array system,the research on the super-resolution reconstruction of camera array is deeply studied in this dissertation,which includes the following research works:(1)A depth estimation method based on energy function minimization is proposed.Image registration is a key step of multi-image super-resolution reconstruction.On the condition that the camera array system is accurately calibrated and the internal and external parameters are known,the scene depth can be used to complete the image registration,and the image registration between all images and the reference image can be realized by the depth estimation of the reference image.In order to fully utilize the comprehensive information of multiple images captured by the camera array system to accomplish the image registration,we propose a depth estimation method based on energy function minimization,and realize the image registration through scene depth estimation.The depth estimation method is verified by experiments and the influence of the number of input images in the depth estimation was analyzed.(2)A depth-based super-resolution reconstruction method using image fusion and blind deblurring is proposed for close-range images captured by camera array.This method decomposes the super-resolution reconstruction process into two steps: image fusion super-resolution and image deblurring.Firstly,the image registration is completed according to the estimated scene depth information,and then the blurred high-resolution image is estimated through the image fusion super-resolution process.Finally the final high-resolution image is obtained by blind deblurring.In the image fusion super-resolution step,the depth information is used to guide the fusion process.In the image deblurring step,as the blur is unknown in the imaging process,we deal with blind deblurring by alternately optimizing the image and the blur kernel.The experimental results show that the proposed method can reconstruct high-resolution images with greatly improving subjective visual effects and objective evaluation indicators when dealing with blurred close-range images,while it reconstructs high-resolution images with good objective assessment and over-deblurring and over-sharpening phenomena in subjective visual effects when dealing with clear close-range images.(3)As the two-step super-resolution reconstruction method based on depth estimation cannot deal with clear close-range images well,a depth-based super-resolution reconstruction method with joint-optimization of image fusion and blind deblurring is proposed.This method transforms the camera array based super-resolution reconstruction problem into a special multi-channel blind deblurring problem by constructing an integrated energy function for estimating high-resolution image and blur kernel,and simultaneously realizes fusion super-resolution and image deblurring through minimizing the integrated energy function.The experiment results have indicated that the reconstruction results are greatly improved in subjective visual effects and objective evaluation indicators when the method is used to process clear close-range images,and the phenomenon of over-deblurring and over-sharpening is mitigated.(4)Two super-resolution reconstruction methods based on optical flow are proposed for long-range images captured by the camera array system.One of the proposed super-resolution methods is a two-step method which is suitable for blurred long-range images,while the other is a joint method which is applicable to clear long-range images.The main difference between the optical flow based reconstruction methods and the depth estimation based reconstruction methods is the image registration.The depth range of long-range images is difficult to determine,the optical flow based reconstruction method estimates the sub-pixel displacement between the long-range input images and the reference image by the optical flow algorithm to complete the image registration.The two-step super-resolution method based on optical flow is similar to the two-step method based on depth estimation in which the super-resolution reconstruction process is decomposed into two steps: image fusion super-resolution and image deblurring,while the joint super-resolution method based on optical flow transforms the super-resolution reconstruction problem of camera array into a simpler multi-channel blind deblurring problem,and constructs a new integrated energy function.Minimizing this integrated energy function through alternately optimizing high-resolution image and blur kernel simultaneously realizes the two processes of fusion super-resolution and image de-blurring.