Micro-Expression Recognition Based On RoI Optical Flow Features

The transmission of emotional information is an essential part of human communication,and facial expression is one of the main ways to transmit emotional information.Facial expressions can be divided into macro-expression and micro-expression.Compared with macro-expression,micro-expression are uncontrollable facial muscle movements that are difficult to suppress,have short duration and weak movement amplitude,and can reflect people’s real emotions.It has great application value in the fields of human-computer interaction,psychological research,public security,medical care and criminal investigation and interrogation.Due to the locality and weak amplitude of micro-expression motion,this thesis proposes a micro-expression recognition method that uses the optical flow information of the Region of Interest（RoI）as the input data of the network model,and designs and adds a multi-scale attention module to the network.The main work of this thesis is as follows.Aiming at the locality problem of micro-expression motion,a micro-expression recognition method based on local features and residual network was proposed.Firstly,for the dataset without labeled apex frames,the temporal difference was used to locate the apex frame.Secondly,according to the optical flow information between the onset frame and the apex frame of the micro-expression sequence and the distribution of facial action units in different expressions,3 RoIs were determined in this thesis.Finally,because the apex frame contains more micro-expression information than other frames,the images of 3RoIs are cropped on the apex frame image and used as the input data of the model.Through experiments on Res Net residual networks with different layers,it is found that the shallower Res Net12 designed in this thesis has better effect.Experiments were carried out on local features and global features,and it was found that using local features as the input data of the model was more effective.Aiming at the problem of locality and weak amplitude of micro-expression motion,a micro-expression recognition method based on RoI optical flow features and attention mechanism was proposed.The optical flow images of 3 RoIs are used as the input data of the model,which are sent to the 3 paths of the model,and then the micro-expression are recognized by the feature fusion results of the 3 paths.A multi-scale attention module is designed and added to the network,so that the network can pay biased attention to different channels and spatial locations of the image in the process of extracting micro-expression features.Through multiple sets of ablation study,the effectiveness of using RoI optical flow images as input data,adding residual connections to the network,and adding a multi-scale attention module are proved.Using unweighted F₁ score and unweighted average recall as evaluation indicators,compared with the existing micro-expression recognition methods,the unweighted F₁ score and unweighted average recall of the proposed model Res-MAM reach 0.9080 and 0.9226 on the CASMEII dataset and 0.8508and 0.8535 on the SMIC dataset.The experimental results show that the proposed method can effectively improve the performance of micro-expression recognition,and verify the advantages of the proposed method compared with other methods.