| In order to deal with the demands of the diversified and high definition video services, the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group work together in a partnership known as the Joint Collaborative Team on Video Coding(JCT-VC), and design a new generation video coding standard—HEVC. Compared to H.264/AVC standard, HEVC can save about 50% bit-rate reduction for equal perceptual video quality. However, there are lot of perceptualredundance that can be compressed in HEVC. According to the characteristics of human visual system, this paper exploit the visual saliency to guide the bit allocation algorithm and the RDO algorithm for better visual quality.This article mainly studies the following aspects:Firstly, HVS characteristics, visual saliency and the Context-Aware saliency detection method to calculate saliency map were introduced. And then the significance of visual saliency for video coding was analyzed.Secondly, the rate control algorithm used in the existing HEVC test model-HM16.7 was studied and analyzed. On account of the existing RC algorithm can’t allocate bit suitable, this paper proposed a weighted bit allocation algorithm(WBA) using visual saliency. Experimental results show that Y-PSNR of saliency areas increase 0.59dB and 0.32dB in all intra mode and low-delay P mode, respectively. On the other hand, the whole Y-PSNR and the Y-PSNR of non-saliency area have a slight decline.Thirdly, studied the rate distortion optimization(RDO) in HM16.7, especially analyze the CTU mode decision process. When calculate the cost in CTU mode decision, we can use the saliency weighted to modify the distortion metric in order to select the mode that has smaller distortion with high bit-rate for the saliency area. Therefore, we propose the RDO improved algorithm based on visual saliency by weighted distortion(WD). The result shows that the Y-PSNR of saliency area increase 0.61dB in low-delay P mode, while the whole Y-PSNR and the Y-PSNR of non-saliency area have a slight decline. However, the DMOS values from subjective experiment show that the RDO improved algorithm based on visual saliency improve the visual quality of coded video. |