Research On Micro-Expression Spotting And Recognition

Micro-expressions(MEs)are a special kind of facial expression,appearing when people try to hide their true inner emotions.MEs are very brief,subtle,and cannot be controlled by personal will.For automatic ME analysis,there are mainly two specific tasks,i.e.,ME spotting and ME recognition.ME spotting is a challenging task because even the ME sequences collected in a laboratory environment inevitably contain some irrelevant motion(e.g.,head movement and blinking)and noise.So far,majority of researches on MEs focus on recognition task,while only a few studies have attempted to perform the spotting task.However,for real world applications,ME spotting is the prerequisite of ME recognition and ME recognition is the ultimate goal of ME analysis.Both of them are indispensable.This paper focuses on these two tasks at the same time.At present,sliding window based feature difference analysis model is widely used in most of the existing ME spotting algorithms.However,previous works improved this model at the feature level,but did not consider the applicability of a single sliding window to describe diverse MEs.Moreover,to deal with the high locality of MEs,most of the existing ME recognition algorithms weighted the block-based features by estimated motion intensity,in which expression-unrelated motion and noise are also highlighted.Furthermore,previous works did not consider the relationship between different action areas and the correspondence between combination of action areas and ME categories.Therefore,to address these problems mentioned above,we propose the following two solutions.(1)Duration-Aware and Mode-Aware Micro-Expression Spotting for Long Video Sequences.Considering the durations and transition modes of different MEs fluctuate greatly,we exploit multiple sliding windows of different scales and modes to accommodate to MEs with diverse durations and transition modes.In addition,the proposed voting based aggregation module integrates the difference analysis results of multiple windows and provides a comprehensive judgment about the position of peak frames.On this basis,a simple yet effective interval generation scheme is designed to determine the boundaries of ME intervals for accurate ME location.Superior performance on two publicly available long video ME databases demonstrates the effectiveness and generality of our proposed framework compared to state-of-the-art approaches.(2)Key Facial Components Guided Micro-Expression Recognition Based on First& Second-Order Motion.Considering that the muscle movements of MEs usually occur in a few key facial components and their surrounding areas,we propose to use the semantic segmentation probability maps of several key facial components as prior knowledge to provide a guidance for feature learning.In addition,a components-aware attention module is designed to learn the relationship between different action areas to enhance current expression-related features and suppress irrelevant motion and noise.On this basis,a parallel shallow residual network is proposed as the MER network,whose two branches take first-and second-order motion as input respectively.It reduces the amount of parameters while ensuring that the model has strong feature representation capabilities.Experimental results demonstrate that the performance of our proposed method outperforms most existing ME recognition algorithms.