Research On Realistic Face Video Coding At Low Bit Rate

Human machine emotional interface (facial expression parameters tracking and extraction, facial expression recognition, parameters transmission and high realistic synchronized speech facial animation) is a hot topic of research in the field of Computer Vision & Computer Graphics and has a lot of applications in Human-Computer Interfaces, Video Coding, Entertainment, and Virtual Reality, etc. In the past 30 years, great progress and developments have been made in these areas. However, at present, it still has a lot of problems. Therefore, how to obtain correct facial motion and expression parameters quickly from video containing face on transmitter, how to transmit these parameters specially using human facial knowledge, how to obtain synchronized speech driven high realistic facial animation according these parameters on receiver, and how to obtain high rate of expression recognition result are challenges, they concern many problems including the motion analysis in computer vision, facial expression recognition, source and channel coding, the kinematic and dynamic modeling and representation of individualized face, the mechanism of co-articulation and text driven facial animation, etc.Facing to ultra-low bitrate model based facial video coding/decoding area, in this paper, we study human machine emotional interface related problems in some aspects, and pay more attention to the issues of facial expression parameters tracking and extraction, parameterized video coding, and high realistic synchronized speech facial animation specially.The innovation aspects and majoy work in this paper are as follows:(1) A face adaptation algorithm based on single image is proposed. Firstly, the first frame containing face in input video is detected. Based on this frame, improved SVM (Support Vector Machine) is utilized for face detection, Adaboost+Camshift+AAM (Active appearance model) are utilized for feature localization. Then the coder gets FDP (Facial Definition Parameter) through Face adaptation of a simple universal triangular model. Finally the decoder adapts a complex universal triangular model using these FDP.(2) A 3D facial expressional motion tracking algorithm based on online model adaptation and updating is proposed. The algorithm constructs the online model using an adaptive statistic observation model, and statistic search and determinately search are applied to observation scene simultaneously using the combination of adaptive state transition model and improved particle filter. Multi-measurements are infused to decrease lighting influence and person dependence. Then, not only global rigid motion parameters can be obtained, but also local non rigid expressional parameters can be obtained.(3) Based on deeply research on facial expression recognition, an algorithm for static facial expression recognition is proposed firstly, facial expression is recognized after facial actions are retrieved according to facial expression knowledge based on particle filter. Coping with shortage of static facial expression recognition, an algorithm combining static facial expression recognition and dynamic facial expression recognition is proposed, facial actions as well as facial expression are simultaneously retrieved using a stochastic framework based on multi-class expressional Markov chains, particle filter and facial expression knowledge.(4) An algorithm for compressing MPEG-4 facial animation parameters (FAP) is proposed. Facial action basis function (FBF) are used to group FAP, then we can lower bit rate by combing intraframe and interframe coding scheme, and it does not introduce any interframe delay.(5) A 3D facial expression animation algorithm based on MPEG-4 is proposed. This algorithm produces facial animation combing parameterized model and muscle model, and can produce high realistic facial expression animation with FAP flow. Furthermore, this algorithm could produce facial viseme actions considering the co-articulation effect in speech. Then according to phonemes from text analysis, phoneme duration, additional expression information, and interpolation between viseme using NURBS, synchronized speech facial expressional animation are obtained.(6) According to the above researches, internationally for the first time, a facial expression parameters tracking and extraction, facial expression recognition, parameters transmission, high realistic synchronized speech facial animation Demo System is constructed. The system could produce high realistic facial animation from decoded parameters on decoder.