Research On The Generalization Of No-referencce Image Quality Assessment Models For Authentic Distortions

Image quality assessment(IQA)plays a vital role in many image processing fields,such as image acquisition,compression,enhancement,generation,and retrieval.Therefore,IQA has attracted the attention of researchers.In the field of quality evaluation,no-reference IQA for authentic distortions has the widest application scenarios.However,the complexity of authentically distorted images makes it very challenging to design corresponding evaluation algorithms.In recent years,the advanced deep nerve networks were applied to IQA,which have made great progress in the evaluation of authentic distortions.However,the deep neural network-based IQA algorithms still have the problem of poor generalization ability,which is manifested in four aspects:(1)Current IQA models cannot commendably describe both local and global aspects of images at the same time,which affects their prediction accuracy and generalization ability;(2)Current IQA models cannot effectively deal with the challenge of content variation,which affects their generalization ability;(3)Current IQA models cannot satisfactorily evaluate low-quality images,which makes it difficult to be adopted to scenes with many low-quality images;(4)Current IQA models have poor performance when the number of training samples is significantly reduced,which makes it difficult to expand to scenes with few training samples.In this thesis,we propose corresponding solutions to solve the above generalization challenges.The main research contents of this thesis are summarized as follows:(1)A no-reference IQA metric by aggerating both local and global deep features is proposed.Previous studies have demonstrated that local and global features play complementary roles in quality assessment,and both of them are essential for IQA algorithms.Existing quality evaluation metrics typically generate network inputs by cropping image patches or resizing images to a fixed resolution.However,these strategies cannot efficiently fuse both local and global features of images,and they cannot fully explore the relationship between local and global features either.Motivated by the above facts,this thesis proposes a new authentic distortion-oriented IQA metric by adopting a visual Transformer to integrate both local and global deep features.The first step of this method is to process the input image and achieve both multi-locational local distortion image and global content image.Then,these two images are sent into the model to extract complementary features.Meanwhile,the proposed algorithm adopts a visual Transformer to mine the relationship between local image patches and the quality of the entire image,and a self-attention mechanism-based module is adopted to further mine the relationship between local and global deep features.At last,the image quality score is predicted based on the fused feature.Extensive experiments performed on five authentically distorted IQA datasets demonstrate that the proposed algorithm outperforms previous methods in terms of both prediction performance and generalization ability.(2)An intelligibility-driven no-reference IQA algorithm is proposed.Two fundamental challenges of IQA affect the generalizability of IQA algorithms,which are the variation of image content and the diversity of image distortion.However,current IQA algorithms mainly focus on assessing the distortion of images and do not fully investigate the variation of image content.Motivated by the above facts,a new algorithm is proposed in this thesis.The algorithm introduces the intelligibility of image content to deal with the challenge of content variation,so as to construct a highly generalizable image quality evaluation model.In this thesis,the first step is to analyze the relationship between image intelligibility and image quality.Then,a bilateral network is proposed to integrate both aspects of image distortion and image intelligibility.During the fusion process,the proposed algorithm further designs feature selection strategies to avoid negative transfer.This method not only extracts traditional distortion features but also adaptively integrates intelligibility features,so as to build a highly generalizable no-reference IQA metric.In this thesis,we perform extensive experiments on five intelligibility tasks,and the experimental results show that the proposed method outperforms state-of-the-art algorithms,and the five intelligibility tasks consistently improve both prediction accuracy and the generalization ability of IQA models.(3)A no-reference quality assessment algorithm based on intermediary-enhanced images and an iterative training strategy is proposed to alleviate challenges of distribution shift and long-tailed distribution in low-quality image assessment.The performance of current IQA models in evaluating low-quality images is significantly worse than that of medium-/high-quality images.Since there are many low-quality images in the real world,poor performance on low-quality images limits the application of quality assessment algorithms in many scenes.In this thesis,we first discover that two challenging problems significantly degrade the model’s performance on low-quality images,which are distribution shift and long-tail distribution.Based on this,we propose a bilateral network based on intermediary enhancement and iterative training to address these two challenges.Inspired by transitive transfer learning,the proposed algorithm adaptively employs quality-enhanced intermediary images to introduce more information about low-quality images for alleviating the distribution shift challenge.At the same time,the proposed algorithm also adopts an iterative training strategy to deal with the long-tailed distribution problem.It decouples feature extraction and score regression to achieve better feature representation and better score regression.This iterative training strategy not only transfers the knowledge learned in the first stage to the second stage but also allows the model to pay more attention to the tailed low-quality images.In this thesis,extensive experiments are performed on five authentically distorted IQA datasets.The results demonstrate that the proposed algorithm not only significantly improves the evaluation performance on low-quality images but also achieves the best prediction performance on the whole dataset.During cross-dataset tests,the proposed algorithm also achieves the best generalization performance.(4)A knowledge-guided no-reference IQA algorithm is proposed to improve the prediction accuracy and generalization ability of the IQA model trained with few samples.Training neural network-based IQA models usually requires a large number of labeled samples,but quality labels are expensive to obtain.Therefore,it is urgent to propose an IQA model with strong generalization ability,which can be trained with fewer labeled samples.Motivated by the above facts,this thesis proposes a knowledge-guided IQA algorithm by integrating both knowledge of the human vision system(HVS)and natural scene statistics(NSS).Specifically,the proposed algorithm first extracts quality-aware HVS and NSS features as prior knowledge.Then,these two types of knowledge are embedded into the deep neural network by training the network to predict the HVS and NSS features.After that,knowledge-enhanced quality features can be obtained,and the final quality score can also be predicted based on this feature.In this thesis,we perform extensive experiments on five authentically distorted datasets and make comparisons with state-of-the-art algorithms.The experimental results prove that introducing knowledge significantly reduces the dependence on the number of training samples,and the proposed knowledge-guided model achieves better performance in terms of both prediction accuracy and generalization ability.