Research On Emotion Recognition Of Different Modalities Based On Facial Expressions And EEG

With the rapid growth of artificial intelligence,affective computing as a branch field is also explored continuously by researchers.Affective computing has three research directions:recognition,expression,and decision.The research on emotion recognition is more extensive and in-depth.Emotion recognition mainly studies how to make the machine recognize human emotion accurately and eliminate uncertainty and ambiguity.Human emotion expression is diversified,which can be achieved by non-physiological signals such as facial expression,speech,text,or physiological signals such as EEG,ECG,and so on.For non-physiological signals,an expression is the most direct way of expression.Therefore,many researchers are committed to the research of emotion recognition based on facial expression,most of which are based on static expression pictures.Considering that expression itself is a dynamic change process,emotion recognition based on dynamic expression video will be more in line with human emotion itself.For physiological signals,EEG signals are closely related to emotion.Due to the spatiotemporal nature of EEG,we can study the characteristics of EEG to improve recognition.But whether emotion recognition is based on expression or EEG,it is a single-mode emotion recognition.Human emotion expression is often accompanied by a variety of ways.For example,when happy,not only face will show a satisfied expression,but also the body has posture,and tone of speech will be higher.Therefore,multimodal emotion recognition conforms to the way of human expression and improves the diversity of its features,which is conducive to the further research of emotion recognition.The main tasks in this paper are as follows:Twelve classic video clips were selected to induce emotion,four of which were positive,neutral,and negative.Through the existing equipment in my laboratory,15 subjects were collected to express and EEG data.Regarding facial expression video,this paper studied emotion recognition based on facial expression.According to the different processing methods of original expression video data,it was divided into emotion recognition based on static expression frames and dynamic expression sequences.The difference between the two is that the former is to process the individual expression frame,and the latter is to process frame sequences after dynamic expression video segmentation,and finally,get the classification results.For emotion recognition of the latter,this paper chose three different methods:frame aggregation,optical frame flow,and frame timing.Compared with static frame recognition,it was found that CNN expression recognition accuracy based on frame aggregation was the highest,reaching 96.2%.About collected EEG data,this paper studied emotion recognition based on EEG.In this paper,three different models were established:single-channel LSTM model,dual-channel LSTM model,and CNN model.The input of the single-channel LSTM model could be divided into five categories:5ms EEG data,10ms EEG data,and three features(variance,energy,and differential entropy)after feature extraction on wavelet transform.The input of the dual-channel LSTM model could be divided into three categories,including two combined features after wavelet transform.Finally,it was found that 10 ms EEG data classification result of a single-channel LSTM model was the best.Compared with single-mode emotion recognition,multimodal feature fusion recognition can make better use of features diversity and correlation between features,to improve its final accuracy.Considering the combination of the two,using the fusion of face and EEG to carry out bimodal emotion recognition,complete classification about three kinds of expression,including positive,negative,and neutral.Therefore,this paper studied multimodal fusion recognition of expression and EEG data.The face-EEG emotion data set was established.Then multimodal fusion methods were introduced and compared.Finally,feature fusion on CNN-LSTM and LSTM-CNN based on deep learning was selected.The results show that the accuracy of the LSTM-CNN model is 93.13%,which is more suitable for fusion classification of expression and EEG.Its recognition accuracy is much higher than that of single-mode EEG emotion recognition in this paper.Still,to some extent,it is lower than that of single-mode expression recognition.It shows that EEG features in this paper interfered with the expression ability of face expression features to some extent.Therefore,more in-depth research on EEG and expression feature fusion is needed in the future.