Research On Motor Imagery Brain-computer Interface Based On EEG-fNIRS

Brain-computer interface(BCI)is a method to realize the direct interaction between brain and computer or equipment by detecting specific activities of the central nervous system and converting the activities into corresponding instructions.BCI allows people to get rid of the dependence on peripheral nerves when interacting with the external environment.In recent years,BCI has been widely applied in rehabilitation field and many non-medical fields.Comparing with other types of BCI,the advantage of motor imagery BCI lies in that it is an asynchronous BCI that does not rely on external stimuli,and can be used at any time in daily life,while other types of synchronous BCI require users to run stimulus program in advance and keep still during task-state.But at present,motor imagery BCI is unable to obtain stable and effective classification results in different subjects,thus it is still hard to be put into practical applications.Electroencephalograph(EEG)has affordable cost and high temporal resolution,is the most extensively used brain signal modality.However,because EEG collects the cortical electrical activity,the electrical signals generated by the muscles and blinking in daily life will be conducted to EEG electrodes and drown out EEG signal.Therefore,using EEG as the signal acquisition method of motor imagery BCI cannot fully utilize the advantage of motor imagery BCI that can be used anytime.Using multi-modal signal as the signal acquisition method is the trend of motor imagery BCI research.Since functional near-infrared spectroscopy(f NIRS)is robust to interference,portable,and relatively affordable,this study chose EEG-f NIRS bimodal signal as the signal of motor imagery BCI,and conducted the following researches:(1)In order to analyze whether the discriminant information contained in the brain network characteristics of EEG and f NIRS are complementary,and to provide a basis for subsequent research,this study constructed functional connectivity networks based on EEG and f NIRS signals and analyzed the differences within them.By comparing the differences in the functional connectivity between different task conditions,as well as the difference in the functional connectivity between EEG and f NIRS under the same conditions,and exploring the differences in the functional connectivity between different brain areas from the perspective of electrophysiology and hemodynamics,the complementarity of the brain network characteristics of EEG and f NIRS,and the rationality of using brain network as the preliminary feature extraction method were investigated.The study found that there are significant differences(p<0.05)in EEG and f NIRS functional connectivity strength of the node pairs involved with the channels in motor cortex between different motor imagery task conditions.EEG has more significant differences(p<0.05)in the connectivity strength of the node pairs involved with the channels in right motor cortex and the brain network metrics of the channels in right motor cortex between different motor imagery task conditions,while the area of f NIRS where has more significant differences(p<0.05)is left motor cortex.In addition,under the same task conditions,the overlapping part of EEG and f NIRS channels were used as nodes to construct brain networks,and significant differences(p<0.05)have been found in the connectivity strength of most node pairs and the brain network metrics of most channels between different signal modality.It can be concluded that the discriminant information contained in the brain network characteristics of EEG and f NIRS are complementary,and using brain network as the preliminary feature extraction method is rational.(2)In order to develop a more rational method to fuse EEG and f NIRS features,this study proposed a graph convolutional network(GCN)based feature fusion method which integrates the connectivity and topology relation between different brain area and different modalities.The performance of the GCN based method was compared with the performance of fusing the features directly in fully connected layer.The classification results show that the average accuracy of EEG is 75.32%,and that of f NIRS signals is 73.05%.When directly using the fully connected layer to fuse different modality features,the average accuracy of the hybrid signal is 76.65%,which is not significantly different from the average accuracy of EEG.When GCN is used to fuse different modality features,the average accuracy of hybrid signal is 79.57%,which is significantly higher than the result of EEG(p<0.001)and f NIRS(p<0.001)and the fully connected layer fused hybrid signal result(p<0.01).(3)In order to develop a more robust model for motor imagery data feature extraction and classification,this study used dynamic functional connectivity(d FC)as the feature,and used long short-term memory(LSTM)as classifier to take full advantage of the discriminant information contained in the temporal variation of d FC and classify the signal under different motor imagery task conditions.The performance of the model was compared with the traditional common spatial pattern(CSP)algorithm.The classification results show that the average accuracy of hybrid d FC is 90.03%,and the average accuracy of EEG and f NIRS d FC are 86.29% and 81.55%,respectively.The average accuracy of hybrid CSP is 53.42%,EEG is68.21%,and f NIRS is 51.60%.Compared with the hybrid CSP method,the average accuracy of the hybrid d FC method is significantly improved(p<0.001),while there is no significant difference compared with the EEG d FC method.Finally,the main results of this study are summarized and divided into the following three aspects:(1)The complementarity of the brain network characteristics of EEG and f NIRS motor imagery signals and the rationality of using brain network as the preliminary feature extraction method are proved;(2)The GCN based feature fusion method has obtained significant performance boost on hybrid signal comparing with EEG(p<0.001)and f NIRS(p<0.001),and the obtained hybrid signal accuracy is higher than EEG accuracy and f NIRS accuracy on each subject;(3)Comparing with the traditional CSP algorithm,the method combining d FC and LSTM has improved the accuracy by 21.82%,and the accuracy of each subject is higher than 80%.