Depth Video Processing Based On Visual Perception

With the advances of the video capture, processing technology and network transmission, as well as the three-dimensional display technology, Three-Dimensional Television (3DTV) has been introduced to the daily life, such as Blu-Ray disc, cable and satellite transmission, terrestrial broadcast, and streaming and downloads through the Internet. Meanwhile, more and more consumers embody a high-quality and immersive multimedia experience via3DTV system, and enjoy high quality visual feast.Multi-view Video plus Depth (MVD), containing scene geometry information, has been the efficient representation format of3DTV and MVV (Multi-view Video) content. The representation method can be utilized to realize auto-stereoscopic displaying at the user end. However, the data amount of MVD is huge. It is necessary to high efficiently compress MVD signals to reduce the overhead of storage and transmission devices, Depth Image Based Rendering (DIBR) technique not only satisfies the demand of stereoscopic display, but also decreases the necessary coded data to some extent. In MVD, depth videos should be transmitted to the user end to aid the virtual view rendering. In fact, due to the limitations of capture technology, the obtained depth videos are not quite accurate. So it is difficult to achieve ideal compression performance, which brought more pressure to the limited bandwidth. In addition, the perception of the Human Visual System (HVS) has drawn more and more attention in the MVD coding, it is imperative to design the depth videos coding algorithm based on the perceptual characteristics. In this paper, we carried out the research of depth video processing based on the visual perception, according to a series of problems in depth video and HVS perceptual characteristics, mainly including the following three aspects:(1) The depth video obtained as compared with the corresponding color video at this stage is inconsisten t in temporal direction, and it is difficult to achieve the desired compression effect. Several depth video processing methods are proposed to solve this problem, which only improve the compression efficiency without combining with the characteristics of the HVS. A novel depth video processing algorithm in temporal direction was proposed in this paper. Motion regions are firstly extracted, and then the motion regions are protected to synthesize the virtual viewpoint. Experimental results show that the proposed algorithm can reduce the bit rate by14.89%～47.48% without the decreasing of the virtual quality.(2) Depth video is used for synthesize the virtual viewpoint. In the process of rendering, the main rendering method is DIBR, the distortion of depth video will lead to the geometric distortion in the synthesized view. The traditional depth video processing method takes into account the linear relationship between the depth map distortion and the geometry distortion in the synthesized view, but does not consider the characteristics of the HVS. By analyzing the HVS, the Just Noticeable Difference (JND) values of the color videos are firstly obtained. Then, combine the relationship between the depth video distortion and virtual viewpoint rendering, the Depth Maximum Tolerated Distortions (DMTD) values of the depth videos are obtained. Considering the inconsistency of the depth videos in the spatial direction and the DMTD values, a depth video spatial consistency enhancement algorithm based on HVS perceptual characteristics is proposed. The experimental results show that the proposed algorithm can reduce the bit rate from10.91%to65.76%while the virtual quality is unchangeable.(3) In the depth video, different regions have rather different effects on the virtual viewpoint synthesizing. A depth perception weighting factor based on HVS is proposed, then the Depth Region of Interest (DROI) of the depth video using the perception weighting factor is extracted. Due to the inconsistency of the depth video in the temporal and the spatial directions, a depth video temp oral-spatial consistency enhancement algorithm is proposed. The main point of the algorithm is to protect the ROI region, and to ensure the virtual quality. Experimental results show that the proposed algorithm can reduce the bit rate from6.39%to37.12%while ensuring virtual quality.