Micro-expression Discriminator Feature Representation And Recognition Based On Deep Capsule Network

Micro expression recognition,as a branch of facial image analysis is an important sub-task in the field of computer vision(CV).It is widely used in national security,judicial and criminal investigation,clinical medicine,teaching evaluation and so on.Micro expressions have a short duration(between 1/25 s and 1/3s),low intensity,and are not easily identified.With the extraordinary progress and development of Internet technology,automatic facial micro expression recognition has become a popular research direction.Currently,there are two types of micro expression recognition methods: traditional methods based on manual feature extraction(such as LBP)and deep learning methods based on neural network(such as CNN,RNN,LSTM).Traditional methods can not extract enough features from the existing time domain and have poor robustness to noise data.In addition to the shortage of training data samples,CNN-based methods of deep learning are difficult to resist the spatial variation of samples.However,Capsule Networks(Caps Nets)replace scalar features with vector representation.This powerful semantic information representation can better express the relationship between entity objects and their parent objects,thus making up for the inadequacy of CNN in resisting spatial changes in samples,and using routing mechanisms to capture more effective facial features to improve micro expression recognition performance.Therefore,based on the capsule network,two micro expression recognition methods are proposed in this paper.The main contributions are as follows:(1)A Micro Expression Recognition(MER)method based on Deep Capsule Adversarial Domain Adaptation Network(DCADAN)is proposed.To alleviate the negative impact of identity-related features,light flow preprocessing was used to encode subtle facial movement information highly related to facial micro expressions.Then,a deep capsule network was developed to calculate the part-whole relationship on the optical flow characteristics.In order to solve the problem of insufficient data and improve the generalization ability of the network,a domain adaptive technology based on the antagonistic discriminator module is proposed,and a macro emoticon dataset is introduced to enrich the available samples.Available samples from macro expression data are integrated into the capsule network to train a fast end-to-end deep network.Finally,a simple and Efficient Attention module(ECA)is embedded in DCADAN to adaptively aggregate the optical flow convolution map into the main capsule layer.The performance of the network on the micro-emoticon baseline crossover database(3DB)was evaluated entirely using the Leave-One Cross Validation(LOSO)analysis.Unweighted F1 score(UF1)and Unweighted Average Recall Rate(UAR)It is used as an evaluation index.The improved MER-based method obtained a UF1 score of 0.801 and a UAR score of 0.829 on DCADAN,which exceeded the previous UF1 score of0.788 and a UAR of 0.782.The comprehensive experimental results show that it is feasible and effective to incorporate antagonistic domain adaptation into the capsule network to represent the discriminatory features of micro expressions.The proposed model is superior to many other deep learning methods for micro expressions.(2)Considering the effectiveness of capsule network in extracting microexpression features,a Dual-Stream Capsule Network(DSCN)micro-expression recognition method based on different modes is proposed.Firstly,the optical flow information between the onset frame and the apex frame is calculated by optical flow method.Then,both the RGB images in the same sequence and the computed optical flow images are respectively fed into two capsule networks to obtain corresponding prediction values,which are recorded as RGB flow and optical flow branches.Finally,the output of the branch of the optical flow is taken as the final result.Specifically,the cross-training is carried out by the proposed deep capsule supervised mutual learning guidance model.The capsule network based on optical flow information is used as the main network,while the RGB image is used as the auxiliary network.In addition,the classification loss is introduced to judge the recognition performance of the network.The divergence loss is used to guide the cross-training and complementary learning between two streams.To validate the effectiveness,the model was evaluated on a mixed dataset(3DB)based on SMIC,CASME II,and SAMM,using the leave-one crossvalidation method.The comprehensive experimental results show that the microexpression recognition performance of this method is better than other deep learning methods.