Multimodal Emotion Recognition Based On The Brain Functional Connectivity Networks

Studies on cognitive science and neuroimaging have indicated that emotion is a complex behavioral and physiological reaction that involves circuits in multiple cerebral regions.However,current studies on multimodal emotion recognition using electroencephalography(EEG)and eye movement data usually utilize single-channel analysis approach to extracting EEG features,which negates the emotion relevant brain functional connectivity networks.This work aims to investigate the emotion relevant brain functional connectivity networks from EEG signals.We present a novel emotion relevant critical subnetwork selection algorithm and extract three brain functional connectivity network features(strength,clustering coefficient,and eigenvector centrality).The results on two public emotion EEG datasets SEED and SEED-V for the three-class and five-class emotion recognition tasks demonstrate the discrimination ability of the brain functional connectivity network features.The best classification performance is achieved by the strength feature,which gains accuracy of 81.53±7.61% and74.05 ± 7.09% on two datasets.Moreover,the strength feature outperforms the state-of-the-art differential entropy feature in classifying five emotions.The performance of the five-class emotion recognition using the 62-channel based brain network feature,eye movement data,and multimodal deep neural network models are 74.05 ± 7.09%,65.21 ± 7.60%,and 84.51 ± 5.11%,respectively.Furthermore,the single modality and multimodal emotion recognition using 18-channel based brain network feature achieve accuracy of72.63 ± 8.26% and 84.45 ± 6.10%,respectively.These results demonstrate the complementary representation properties of the brain functional connectivity network feature and eye movement feature,and the brain functional connectivity network constructed with 18 electrodes has potential in braincomputer interaction systems under real scenarios.Besides,our work also reveals the brain functional connectivity network patterns for the five emotions in the five frequency bands.