Multimodal Emotion Recognition Based On The Combination Of EEG And Eye Movement Signals

Emotion is a very complex psychological state or process of human beings,which can reflex the the perception and attitude of humans and plays an important role in the communication between human beings.The research on emotion recognition is of great importance to the application of human computer interaction(HCI).The environment of HCI system is complicated and dynamic,which requires the system to be able to collaborate with people smartly.A system with the ability of recognizing emotions of human beings can fit better into such complicated situations.If HCI system can accurately detect,recognize and understand human's emotions,the process of human-computer could be much more friendly and harmonious.The EEG signal has a strong representational capacity on the change of brain state of humans,which can be applied to the emotion recognition.In addition to EEG signal,this paper employs another external physiological signal,the eye movement signal,to recognize emotions.This paper focuses on the study of distinguishing ability and characterization of EEG and eye movements signals under different emotion states.This paper combines EEG and eye movement signals by using different modality fusion strategies,including deep neural network,to enhance the accuracy of emotion recognition.Meanwhile,this paper shows the stability of the ability of EEG and eye movement signals to recognize emotions accross different days.Besides,this paper uses several electrodes of EEG signals according to the key regions of the brain related to emotions to recognize emotions successfully,which is of great value to the development of wearable deveices of brain-computer interaction and the practical application of emotion recognition in the fields of health care,transportation,education,military affairs,etc.