| Image restoration is the inverse process of image degradation,aiming to recover the image before degradation,which is an important research topic in the field of image processing.In the past decade,deep learning has made great progress and facilitated the development of image restoration significantly.However,after a series of breakthroughs,it is increasingly difficult for image restoration to further improve performance.In addition,the ill-posed character of image restoration leads to blurred results and poor visual quality,which still exist in deep learning-based methods.How to solve these two problems is an important direction of image restoration nowadays.While the structure is a typical feature of images,which can be introduced as an intermediate variable to regularize the learning process and thus improve the performance.At the same time,structure determines the clarity of images,which provides an approach to solve the problem of blur.From these twofold motivations,we investigate three different image restoration tasks,image inpainting,image denoising and image super-resolution in this thesis,based on the idea of structure inference and assistance.The main work is summarized as follows:We propose an edge-assisted generative image inpainting method,which adopts edge as structure.Based on the idea of structure inference and assistance,we split image inpainting into two steps in the deep learning-based approach,firstly edge generation and then edge-constrained image inpainting.We adopt the edgeness map from a convolutional neural network-based edge extractor as the structure,and propose generating edgeness map of missing regions based on generative adversarial networks(GAN).Our approach of completing structure can deal with more cases than traditional edge-based inpainting method,which restored structure by curve fitting.The completed edgeness map is introduced as additional prior into the subsequent image inpainting network to constrain the restored image,resulting in superior results in terms of both quantitative metrics and visual quality compared to state-of-the-art image inpainting methods.We propose a segmentation-assisted image denoising method,which adopts segmentation maps as structure.Inspired by the classic denoising methods which perform collaborative filtering on similar patches and based on the idea of structure inference and assistance,we propose an approach with two steps.We first estimate the segmentation maps of noisy images,then conditionally normalize the feature maps in the denoising network according to the segmentation maps,resulting in non-local consistency of pixels with respect to semantic categories.To the best of our knowledge,we first improve both the visual quality and signal fidelity of denoising via semantic segmentation.On the other hand,we find that the accuracy of semantic segmentation is deteriorated by noise,while the loss can be partially made up by introducing image denoising before semantic segmentation.With the twofold motivations,we propose a boosting network which alternates denoising and segmentation to enhance their performance.We propose a controllable-edge-assisted real-world image super-resolution method,which adopts controllable edge as structure.Existing real-world image superresolution methods still suffer from blur,even though GANs are adopted.At the same time,the structural artifact is another puzzle of them.To address these two structurerelated problems,we further explore the potential of the idea of structure inference and assistance to solve the composite degradation in real-world image super-resolution.We propose a two-stage pipeline:the first stage estimates high-resolution edges,and the second stage constrains the super-resolution results by the edges.But there are many possible solutions of the high-resolution edges.To reduce the indeterminacy,we propose to control the density of edges by users,so as to guide the estimation of high-resolution edges.With the assistance of edges,we alleviate the problems of blur and structural artifact,and significantly improve the visual quality of real-world image super-resolution.Besides,users control the density of edges and influence the clarity of super-resolved images,leading to diverse super-resolution.In application,users can choose the results according to their preferences respectively.In summary,the general pipeline of structure inference and assistance decomposes an image restoration problem into two steps:first estimating the structure of restoration results from the degraded images,and then using the structure as a constraint to aid image restoration.This thesis adopts the core idea to design and explore algorithms in three classic image restoration tasks.Comparative experiments demonstrate the superiority of the proposed algorithms for the respective tasks.We follow the research process of moving from the particular to the general:image inpainting and image denoising restore the images with single degradation,and real-world image super-resolution deals with composite degradation(composed of blur,noise,compression and downsampling).Through these three different tasks,we validate the generalization and effectiveness of structure inference and assistance in image restoration. |
