Mild Depression Recognition Fusion Model With EEG And Pupil Signals Based On Attention Mechanism

Depression is a common mental disorder,which seriously impairs patients’ ability to cope with daily life.The diagnosis of mild depression can help patients with early intervention and treatment to avoid the increase of severity.At present,there is insufficient attention to mild depression,and there is a lack of objective quantitative evaluation methods.Electroencephalogram(EEG)has been widely used in depression research due to its advantages of economy,safety and non-invasive.Pupil signals can measure emotional arousal and affective processing,and are potential physiological indicators for depression recognition.Multimodal fusion can use the complementary information of multiple modalities to make the depression recognition more accurate and objective.Multimodal data fusion has become a focused issue in current research.This paper attempts to fuse EEG and pupil area signals to establish an effective model for detecting mild depression.Based on the paradigm of free browsing of emotional pictures,EEG and pupil area signals are synchronously collected from 20 patients with mild depression and 20 healthy controls.Based on mutual information and Transformer,two fusion models of EEG and pupil data are proposed to mine the correlation and complementary information of the two signals.The main work and innovation of this paper are as follows:(1)Mutual Information Based Fusion Model(MIBFM).EEG data is a kind of high-dimensional data.In order to explore an effective data dimensionality reduction method and optimize feature space,this paper proposes MIBFM.MIBFM uses mutual information to measure the correlation between EEG and pupil signals,and uses electrode selection algorithm to select a small number of EEG electrodes with the highest correlation with pupil signals.Then,effective EEG and pupil features are extracted in different bands,and the denoising autoencoder is used for fusing bimodal features.Finally,multiple classifiers are used for classification.The results of the experiment demonstrate that the accuracy of MIBFM can reach 87.03% when only 8electrodes were selected.Compared with single-modal features and other fusion methods,MIBFM achieves better classification performance.MIBFM selects a small number of EEG electrodes,which provides a new idea for fusion of EEG and pupil signals,and also provides a theoretical basis for the development of relevant portable applications.(2)EEG and pupil signal fusion model based on Transformer.In this paper,fusion models of EEG and pupil are constructed based on Transformer model and multi-head attention.Firstly,this paper constructs the single-modal models of EEG and pupil based on Transformer,and the classification accuracy can reach 89.75% and 84.17%respectively.Based on the single-modal models,the fusion models are constructed using three strategies,namely standard self-attention,cross-modality attention and attention bottlenecks.The fusion model based on standard self-attention uses the standard Transformer layer to learn the correlation between two modalities.In the fusion model based on cross-modality attention,each modality maintains its own model structure and exchanges information through cross attention.The fusion model based on attention bottlenecks introduces the bottleneck units,and the cross-modal information only flows between single modality and attention bottlenecks.For each fusion strategy,three fusion positions of early,mid and late fusion are tried.The experimental results show that the mid fusion model based on attention bottlenecks can achieve the highest classification accuracy of 93.25%.Transformer can learn the longterm time-dependent relationship between EEG and pupil signals and the correlation between two modalities.In conclusion,to achieve effective recognition of mild depression,this paper constructs two EEG and pupil signal fusion models from the perspective of reducing the number of EEG electrodes and multi-head attention.This paper makes use of the advantages of multimodal data by capturing the correlation and complementary relationship between two modalities,so as to make the recognition of mild depression more accurate and comprehensive.This paper provides technical support for the detection of mild depression based on EEG and pupil signals,and also offers a theoretical basis for the development of related applications.