Research On Micro-Expression Recognition Based On Multi-scale Fusion Capsule Network

Micro-expression(ME)is a kind of spontaneous facial movement with short duration and low intensity,which has a wide application prospect in medical diagnosis,online learning and other fields.In the early stage,facial ME recognition methods were based on traditional machine learning theory,but there are some problems such as low recognition accuracy and poor robustness.With the rapid development of deep learning in the field of computer vision,researchers begin to apply it in ME recognition tasks.Strategies such as increasing the depth,width and residual network of convolutional neural networks are usually adopted to improve the accuracy of ME recognition.However,due to the small number of public samples of spontaneous ME database,the improvement effect of these strategies in the challenge task of ME recognition is limited.In recent years,multi-scale convolution and capsule networks have shown superior performance in extracting and classifying fine features,providing a new direction for ME recognition.Therefore,this paper explores a study of ME recognition based on multi-scale convolution fusion capsule network.In general,the main contributions of our work are as follows:(1)In order to achieve the MEs diversification movement characteristic of spatial relations,a network based on multi-scale convolution integration of multi-channel capsule is developed for ME recognition method.Firstly,the motion information of ME was acquired by the optical flow operator,and the negative effects related to identity information were degraded.Then,ResNet-18 network was exploited to extract the global features of the sequence frames from ME images,and atrous convolution was further used to alter different sensitive fields to enrich the local features of ME.Finally,in order to further explore the subtle spatial relationship of ME features,multi-channel capsule framework for classification and recognition was designed,aimming at improving the feature representation performance of the traditional single-channel capsule network.Three widely used datasets,namely SMIC、SAMM、CASME Ⅱ,and integrated into a cross-database(3DB),were used for evaluation experiment.The experiment result shows that our proposed method is superior to many other ME recognition methods.(2)In view of the powerful feature representation and recognition ability of the capsule network for ME,and aiming at the problems of slight action intensity of ME and small spontaneous open database,this work proposed a multi-branch fusion of attention capsule network for ME recognition.Firstly,the onset frame and apex frame were selected to obtain the optical flow feature information of the ME.Secondly,the convolutional kernel with different sizes of three branches was introduced to extract diverse optical flow features,which were then input into the displacement attention module to enhance the fine features of the ME.Finally,the extracted features were classified and recognized by the capsule network.Experimental results show that the proposed approach can extract multi-level features and improve the accuracy of facial ME recognition.