| In order to obtain lower delay which can bring about real-time interaction,the live streaming system try to give up TCP and use UDP which does not provide reliable transmission as the transport layer protocol.Because transmission errors are unavoidable,error control need be performed at the application layer for high video transmission quality.As a decoder-based error control technology,error concealment has a more direct effect on reducing frame distortion.Error concealment methods can be divided into three categories:based on single temporal or spatial error concealment,based on the combination of temporal and spatial error concealment,based on deep neural network.The traditional single temporal or spatial error concealment method has poor performance.The method based on deep neural network has high computational complexity and can’t adapt to the diversity of transmission errors.The method based on the combination of spatial and temporal error concealment has good performance and relatively low computational complexity,so it is more suitable to be applied to livestream system.The main work and contributions of this paper are as follows:(1)The spatial and temporal error concealment can not utilize characteristics of live streaming data,such as obvious dynamic and static regions in frame content and transmission delay between frames effectively,the paper proposes a spatial-temporal error concealment method based on regional characteristics,which is suitable for live streaming system.The method combines three traditional error concealment methods:motion vector estimation,adaptive homography transform and edge direction interpolation.In order to reduce the processing time of frames,the frame content is divided according to the motion state.Before implementing homography transformation,motion vector estimation which has lower time cost is performed to conceal smooth motion areas.In order to confirm that the macroblocks with high spatial correlation have better visual effect,the macroblocks with poor performance after the implementation of homography transformation are selected by evaluation method,and the spatial interpolation based on edge direction is used for restoration.In order to avoid introducing additional transmission delay,the method only takes the received forward frame as the reference frame;(2)For the unreasonable hyper parameter in adaptive homography transformation,a formula for determining the value of hyper parameters is proposed leading to the adaptation of hyper parameters.The formula fully considers the characteristics of frame content and frame resolution.To relieving from obvious block effect,the boundary matching degree is added as the error evaluation index which can find out more macroblocks with poor performance.Then,edge direction interpolation is performed to conceal these macroblocks;(3)Design and implement h.264 decoder with new error concealment method which can provide stable and reliable decoding ability,and conceal the corrupted frame.The experiment results show that the visual effect of the error concealment method proposed in this paper is almost the same as that of the baseline paper,the computation complexity can be reduced by about 50%.In a nutshell,our proposal makes a good tradeoff on performance and time consumption,and can be applied to the real live streaming scene. |
