Research On Reference Based Methods For Blind Face Restoration

With the rapid development and popularity of capturing equipment,digital image which is usually regarded as the carrier of visual information,has become an indispensable part in people’s daily life and plays an important role in information acquisition,sharing,memorizing and analysis.Among these images,face occupies a high proportion,and is very important in our daily life.For example,taking pictures for people by using smartphone or camera,people in short video or movie,pedestrians in surveillance scenario,face recognition and unlocking related device,etc.However,due to the limitations of hardware devices,unstable factors in the process of capturing,imperfect imaging system,subsequent storage and transmission,it is inevitable to damage the face image,resulting in the decline of visual quality.Real-world low-quality face image restoration,also known as blind face restoration,is a classical research topic in the field of image restoration.Because of its complex and unknown degradation process,which cannot be simulated,it brings great challenges to the restoration task.Blind face restoration cannot only improve the visual perception and reduce the dependence on hardware devices,but also has great significance for subsequent high-level tasks,like face analyses and recognition tasks.Generally,existing restoration methods usually directly learn the mapping from a low-quality image to its corresponding high-quality one via convolutional neural network.In order to improve the restoration performance,researchers usually introduce some prior information such as facial landmarks,semantic labels and so on.However,the purpose of restoration is to generate richer and photo-realistic textures.These existing methods can only obtain limited improvements,and cannot well handle the real-world low-quality face images.In order to improve the performance of blind face restoration,this thesis explores from two aspects,that are generic and specific face restoration,respectively.By introducing the high-quality references of the same identity and large amount of general face texture prior,it can help to reduce the sensitivity to unknown degradation types,and generate photo-realistic textures,which can further improve the performance of blind face restoration.The main research contents and contributions are summarized as follows.(1)Recent restoration methods cannot effectively handle the real-world low-quality images due to the unknown degradation types and usually ignore the identity-aware details for the severely degraded images.Based on this observation,this thesis reformulated this task as specific restoration by introducing a high-quality reference image of the same identity to guide the blind restoration,which can eliminate the sensibility of unknown degradation types and effectively improve the blind restoration performance.Because the reference and the degraded input usually have different poses and expressions,this thesis suggested a semi-supervised manner to predict the optical flow,which is utilized to solve the spatial and pose misalignment problem and subsequently boost the accurate texture transformation.The analyses and experiments demonstrate that compared with other methods,this work can effectively improve the performance of blind face restoration.(2)Although the single reference based method can improve the performance of blind restoration,the improvement is still limited due to the diverse expressions and poses.For example,when the mouth of the reference image is open,and the degraded input is closed,the network cannot effectively utilize the guided information.We find that each person usually has multiple high-quality face images with different poses.Inspired by this observation,this thesis extends the single reference to multiple exemplars for restoration.Given a degraded input,we select the optimal reference with the smallest differences of poses and expressions,which is subsequently utilized through an adaptive feature fusion module to solve the inconsistent illumination distribution.The experimental analyses show that the proposed method can further improve the performance of blind face restoration even handling the input with diverse poses and expressions,mainly because this method has a more consistent reference than these methods with only one reference.(3)The former two specific restoration methods require one or more high-quality references from the same identity.However,the limited number of references with the same identity cannot cover all the poses and expressions,which severely limits the application scenarios.Through careful observation,we find that each person usually has similar structure and texture,so we can always find a suitable image from a large amount of existing high-quality images with arbitrary identity,which has the same and consistent poses and expressions with the degraded input.Inspired by this observation,we suggest a general face restoration method by constructing a general texture prior for each facial component,which can guide the restoration of arbitrary degraded images without requiring references of the same identity.Specifically,given a large amount of highquality face images,we generate the dictionary cluster for each component by utilizing means in the feature space,which is taken as reference for the subsequent restoration.The experimental results and analyses show that with the general high-quality texture priors,which cover nearly all the expressions and poses,this work can effectively and stably handle the blind restoration task and can be used in more wide application scenarios.(4)Although the specific restoration method can transfer the identity-aware texture to the degraded image,and generate more consistent textures with the current identity,it is limited by the consistency of the guided image and the degraded input.When the references are not enough,the performance will degrade obviously.On the contrary,although the general texture prior can cover most of the poses,it often loses the identity related textures.Based on this observation,we propose a dual(generic and specific)memory dictionary method to utilize the similarity and uniqueness of facial structures.Moreover,we can adaptively handle the degraded input with or without reference with a single unified model.Specifically,specific dictionary stores identity related texture features,while general dictionary stores numerous general texture priors.When the identity is unknown and the references are not available,we can also handle it by only using the generic dictionary,which can be flexibly applied in most scenarios(with or without reference).The analyses demonstrate that by combining the advantages of dual dictionaries,our method can generate more photo-realistic and identity-consistent textures for most real-world low-quality images and show practical values in real-world applications.