Deep Learning Based Image Restoration And Editing

With the proliferation of mobile imaging devices(e.g.,smartphones)and the increasing use of digital images as inform The goal of image restoration techniques is to extract clear,high-quality images from degraded images.Traditional image restoration methods mainly focus on the artificial design of priors for clear image,and then finding the optimal result based on maximize-a-posterior method.In recent years,with the development and progress of deep learning technology,it has achieved unprecedented success in the field of computer vision and image processing.Thanks to the powerful modeling ability of the deep learning model,the recent image restoration technology has made a major breakthrough.Based on the latest progress of deep learning,this paper studies and discusses image restoration,including not only traditional image restoration problems,such as denoising and deblurring,but also image restoration from sparse information,such as image generation based on semantic segmentation mask.The main research results mainly include three aspects:(1)Based on the denoising diffusion model,a progressive face image denoising algorithm is proposed in this paper.The traditional image denoising methods based on maximum a posteriori(MAP)usually use the artificially designed image prior,which can not model the real distribution of images.In order to solve this problem,this paper uses the recently proposed denoising diffusion generative model to represent the image prior to solve the image denoising problem.The denoising diffusion model has a good performance in image generation.The model takes the noisy image and time as inputs to predict the noise in the image,so as to gradually generate the image.Denoising diffusion models estimate the distribution score of noisy images at different times,and the distribution score is the gradient of the logarithmic probability density function.This paper uses the image prior learned from the diffusion model and the gradient based optimization method to solve a MAP problem.In this paper,a pre-trained denoising diffusion model based on face dataset is used to denoise both facial images and natural images.The proposed method can achieve clearer and more detailed denoising results.And this article also analyzes the underlying reasons why the model performs well on non-face and non-Gaussian noise data.(2)Using deep neural networks to learn both the prior and the data fitting term in the image restoration problem is used for non-blurring of images.Most of the non-blurring methods focus on designing or learning image priors that tend to be clear and where the image noise obeys a Gaussian distribution.When it comes to real-world deblurring,where the noise is non-Gaussian and the blur kernel is inaccurate,these methods will not be able to handle such cases.In this paper,we propose a deep learning framework that learns both data fitting and image prior.Thanks to the powerful modeling capability of deep neural networks,the model contains a data fitting network and an image prior network that can learn complex distributions and extract the internal associations of images.Due to the presence of special operators,this paper also provides methods to compute the gradients and parameters of intermediate results,which provide a basis for network training and will inspire the development of related works.The method is evaluated on a general dataset and a synthetic dataset,and experimental results show that the method works better than existing methods for image deblurring tasks.(3)Recovering and editing images from semantic mask information.Semantic segmentation mask refers to the information image after semantic segmentation of the image.Generative adversarial networks conditioned on sparse information such as semantic masks have achieved great success in face image generation.However,inconsistency may arise in the appearance of different regions during regional editing of faces.To address this issue,this paper explores the harmonization problem of face regional style transfer and editing for the first time.First,a multi-level encoder is proposed to extract the deep encoding of the appearance in different facial regions,and a style mapping network is used to generate random regional style codes from random noise.Then,a multi-region style attention module is proposed to adjust the style codes from a reference image region to fit the overall style in the target image,with the goal of generating a more harmonious image.Furthermore,an invertible flow model is used to fine-tune regional styles.Finally,the proposed model is evaluated on three widely used face datasets,and the results show that the model performs better than previous methods in terms of image quality and harmony degree.The method can also be used for interactive editing of facial semantic region shapes and colors.