Emotion Recognition Based On Visual Information And EEG Information Fusion

Emotion contains important information in the process of people’s communication,and the influence of different emotional states on people’s decision-making and perception is also different.In recent years,with the development of artificial intelligence science,endowed the machine with emotional intelligence to achieve human-computer interaction is also a research hotspot in the field of artificial intelligence.A multimodal emotion recognition method based on the fusion of visual information and EEG information is proposed.The research work includes the following several aspects:(1)Emotion recognition based on facial expression imagesAdaboost algorithm based on Haar feature is firstly used to face detection and clipping of image samples to remove irrelevant information interference outside the face region.Then we use the deep neural network built by ourselves and the deep neural network trained by transfer learning to recognize facial expression,and compare the two methods.Finally,using the method of secondly finetune the VGG16 model,the recognition effect of the expression recognition model on the spontaneous expression is improved.The experimental results show that the proposed method is robust to the recognition of spontaneous emotions.(2)Emotion Recognition Based on EEG SignalThe original EEG signal is denoised,then,the EEG signal is divided by wavelet transform and reconstruction and the differential entropy feature of rhythm wave is extracted.Finally,the SVM classifier is used to classify the differential entropy features of different rhythm waves to train the emotion recognition model.The experimental results show that the EEG contains a lot of emotional information,which can accurately reflect the emotional state of the human.(3)Multimodal Emotion Recognition Combining Facial Expression and EEG SignalThe attention model is used to fuse the facial expression recognition results and the EEG emotion recognition results in the decision layer.Firstly,the emotion recognition results are obtained by facial expression recognition model and EEG emotion recognition model,and then the weights of two recognition results are adjusted by attention model to establish multi-modal emotion recognition based on facial expression and EEG signals model.The experimental results show that the accuracy of multimodal emotion recognition with fusion of facial expression and EEG is higher than that of single modal emotion recognition.