Research On Micro-Expression Recognition Based On Improved Capsule Network

Micro-expressions are brief and involuntary rapid facial emotions that can reveal a person’s true emotion and have a wide application prospect in the fields of criminal interrogation,clinical diagnosis,national public security,and so on.Compared to typical macro-expressions,micro-expressions are short in duration(typically only 1/25 s to 1/5s),low in intensity(muscle movements only occur in small areas of the face),and less easily identified.With the advancement of computer vision technologies,automatic facial microexpression recognition has emerged as a significant research area.Nowadays,there are two primary categories of micro-expression recognition methods in common use: traditional methods based on manual features and deep learning approaches based on convolution neural networks(CNN).Among them,the traditional methods mainly rely on the quality of hand-designed features,which are not robust to noisy data and have a low classification rate.Compared to traditional micro-expression recognition methods that rely on manual feature descriptions,the deep learning method using CNN integrates feature extraction and classification automatically in an end-to-end manner to achieve better recognition performance in the field of micro-expression recognition.However,CNNs do not consider the structural information of faces and require a large amount of data for training.Capsule networks(CapsNets)are proposed to solve the problems of classical CNNs,which can improve the performance of micro-expression recognition by considering the relationship between the part and the whole of the feature,using vector representation instead of scalar features,and using dynamic routing mechanism to extract more adequate and effective highlevel features.However,the capsule network also has some drawbacks.It only uses two convolutional layers to extract image features which is not suitable for classification tasks of complex images,and there is no clear channel dependency relationship within the modeling module in the original capsule network.Therefore,this thesis aims to study the improved capsule network and apply it to micro-expression recognition.The following are the details of the research for this thesis:(1)An improved capsule network called RES-CapsNet is proposed to overcome the limitations of the original Capsule Network.Firstly,the Res2 Net block is applied to enhance the convolutional layer’s feature extraction capability,extract multi-scale image features,and increase the sensitivity field of each convolutional layer without involving a large number of parameters.Secondly,the squeeze-excitation(SE)block is added into the Primary Caps to eliminate the effect of the original capsule network which tends to interpret all the features in an image on the image classification task.By distributing different weights to different feature mapping channels,the SE module can highlight useful features and suppress useless ones.Finally,an effective channel attention module,the ECA module,is used between the convolutional layer and the Primary Caps.Although the ECA module only involves a few parameters,significant performance gains can be achieved.In order to assess the effectiveness of the proposed approach,LOSO cross-validation is performed on three spontaneous microexpression datasets and the composite dataset.The improved capsule network RES-CapsNet achieves UF1 score of 0.7419 and UAR score of 0.7458 on the composite dataset,exceeding the baseline capsule network(UF1-0.6520,UAR-0.6506).Extensive experiments show that our proposed method can extract more micro-expression features,which demonstrates the effectiveness and robustness of the improved capsule network.(2)Based on the validation that the capsule network is effective for micro-expression feature extraction,a lightweight improved capsule network called PPLC-CapsNet is proposed.The network uses the backbone network of PP-LCNet for feature extraction,which drastically reduces the number of parameters of the network and can facilitate the network to acquire stronger feature representation without increasing the latency.In addition,a parallel-type polarized self-attention block is added to the Primary Caps to reduce the information loss caused by dimensionality reduction.The proposed architecture first performs data preprocessing on the input micro-expression sequences to obtain apex frames and then establishes a PPLC-CapsNet model to efficiently extract micro-expression features and classify them.To verify the effectiveness of the PPLC-CapsNet in micro-expression recognition,the performance of the method is tested on CASME II,SMIC,SAMM,and the composite datasets using the LOSO cross-validation method.The experimental results show that the method achieves higher recognition accuracy than the baseline capsule network with a low number of parameters,and can obtain more excellent performance on the classification task of micro-expression images.