Research On Image And Video Quality Assessment Via Feature Learning

With the rapid surge in the development of high speed broadband network,the commercial use of 4th generation mobile network and the growing amoun-t of intelligent terminals, enormous amount of visual media content has been brought into our daily life. Since human visual system (HVS) are sensitive to vi-sual signal impairments, e.g., blockiness, blurriness, noisiness and transmission loss, it is crucial to guarantee high quality of experience (QoE) for consumer-s. Evaluating the influences of various distortions on the perceived image and video quality through a quantitative approach is a fundamental and challenging task. Generally, visual quality assessment metrics can be adopted in the fol-lowing four categories of applications: 1) tuning parameters of digital devices;2) benchmarking the image and video processing systems; 3) monitoring signal perceptual quality in the visual communication network and allocating network resources; 4) ranking photos for image recommendation systems. Subjective quality evaluation is the most accurate and reliable approach for visual quality assessment, but it is time consuming, expensive, non-reproducible, and unable to be implemented in real world systems. Therefore, automatic objective im-age and video quality assessment methods which are consistent with human perception are highly desirable.In this thesis, some basic problems about visual quality assessment are ad-dressed. Based on the development of feature learning knowledge in computer vision and machine learning domain,this thesis investigates to design fast and efficient no-reference (NR) video quality assessment (VQA) and image quality assessment (IQA) methods. In addition, this thesis also studies efficient statis-tical metric fusion methods to automatically enhance full-reference (FR) IQA methods. The main contributions of this thesis are summarized as follows:(1) A novel “opinion free"(OF) general purpose NR VQA method based on frame-level unsupervised feature learning is proposed. The system consists of three components: feature extraction with max-min pooling, frame quali-ty prediction and temporal pooling. With a codebook from normalized image patches, frame level features are extracted by unsupervised feature learning and used to train a linear support vector regression (SVR) for predicting quality s-cores frame by frame. Frame-level quality scores are then combined by the temporal pooling which is inspired by HVS to obtain a single video quality s-core. Experimental results on two video quality databases show that without training on human opinion scores the proposed method is comparable to state-of-the-art NR VQA methods.(2) Two fast novel general purpose NR IQA methods based on local patch information aggregation are developed. Both methods extract normalized raw image patches as local features and require very small codebook to learn the global quality aware feature. The first method, LGFA, constructs a codebook via Gaussian mixture model (GMM) clustering and each local feature is as-signed to all codewords. Then the gradient of feature log-likelihood on GMM which represents the relationship between local features and codewords is cal-culated. The second method, HOSA, constructs a comprehensive codebook via K-means clustering. In addition to the mean of each cluster, the diagonal co-variance and coskewness (i.e., dimension wise variance and skewness) of clus-ters are also calculated. Each local feature is softly assigned to several nearest clusters. Then the differences of high order statistics (mean, variance and skew-ness) between local features and corresponding clusters are softly aggregated to build the global quality aware image representation. Both methods utilize SVR to learn the mapping between perceptual features and subjective opinion scores.The proposed methods have been extensively evaluated on several large-scale image databases and shows highly competitive performance to state-of-the-art NR IQA methods with only 100 codewords. Not only natural content images can be well evaluated, but also document images and screen content images which contain natural scene, text and graphics together are well evaluated. And images with authentic distortions are also estimated.(3) A novel general purpose NR IQA method based on image pixel in-formation aggregation (PIA) is designed. The local binary pattern (LBP) on raw images is introduced in this method and the global quality aware feature is learned by image pixel information aggregation. The signs of local pixel dif-ferences are computed as well as the magnitudes of local pixel differences are incorporated. In addition, the pixel differences information in a perceptual col-or space is also considered. Experimental results on two large image databases demonstrate the superiority of the proposed method. The new method can deal with both image quality prediction and image distortion identification prob-lems. Furthermore, images with chromatic distortions are better evaluated than previous methods.(4) Two novel statistical metric fusion (SMF) methods for FR IQA metric enhancement are proposed. First, local quality map is constructed from exist-ing state-of-the-art FR IQA methods. After that several statistical indices are extracted from local quality map. Finally, the extracted statistical indices are fused by supervised statistical metric fusion (SMF-S) based on SVR and un-supervised statistical metric fusion (SMF-U) based on reciprocal rank fusion(RRF) to obtain the final quality score, respectively. Experimental results on the largest public IQA database have demonstrated that the two proposed SMF methods can generally enhance the quality prediction performance of FR IQA methods in terms of high correlation with human opinion scores.