Research On The Application Of Multi Stream Convolutional Neural Network In Brain Computer Interface Based On Motor Imagery

Brain-computer Interface(BCI)is a communication system independent of the normal output pathways composed of peripheral nerves and muscles,which provides a new way to realize brain-computer interconnection.Brain-computer interface system based on motor imagery is considered by many researchers as the most development potential of a braincomputer interface system,it can help people directly by thinking to control the robot based on BCI interfaces,which makes the brain-computer interface is not only in the field of rehabilitation for the disabled and elderly care medical has significant advantages,but also in education,military,entertainment,intelligent household and so on also has a broad application prospect.Electroencephalography(EEG)contains physiological information related to movement.Learning this part of information from the original EEG can provide great help for the recognition of motor imagery EEG.With the advent of the era of artificial intelligence,deep learning has become one of the most popular scientific research trends today.In particular,Convolutional Neural Network(CNN)has achieved remarkable achievements in image recognition,speech processing and natural language processing.Compared with manual selection of EEG signal features,convolutional neural network can integrate feature extraction and feature classification into a network,realize end-to-end learning of EEG signals,and retain the feature information of original EEG signals to the maximum extent.In addition,the current research can be divided into time domain analysis,frequency domain analysis,time-frequency analysis and space domain analysis according to the observation level.Most researchers use a single analysis method,so the extracted features can only reflect one aspect of the EEG signal,rather than the whole brain activity.In order to solve the above problems,a multi-stream convolutional neural network model is proposed in this paper,which can simultaneously extract the EEG features in the time domain,frequency domain and spatial domain,and fuse the various features in a specific way,and then automatically classify them.The experimental results show that the proposed multi-stream convolutional neural network has a good classification result for EEG signals based on motor imagery.The research content of this paper mainly includes the following two aspects:(1)Time-frequency dual-stream convolutional neural networkIn the research of BCI,the characteristics of time domain and frequency domain are the two most commonly used features.In order to fully use of time domain and frequency domain characteristics of the EEG signals,we put forward a dual-stream convolution neural Network(DCNN),when the model contains two independent CNN,respectively using time domain and frequency domain signal as the input signal,after convolution pooling study EEG characteristics of time domain and frequency domain characteristics,then the time domain and frequency domain characteristics of linear weighted fusion,the final classification results are obtained.In this model,the weight fusion coefficient ? can be automatically learned by DCNN,so as to find the best weight coefficient value for each subject.In addition,this study discusses how to improve the classification performance of EEG signals by comparing different fusion methods.Finally,experiments are carried out on BCI2008 dataset 2a to explore the influence of weight coefficient ? on the recognition accuracy,the influence of the number of neurons in the fusion layer on the recognition performance,and the influence of the number of network layers on the recognition performance.(2)Time-frequency-space three-stream convolutional neural networkFor multi-channel EEG data,there is a certain spatial information in the distribution of sampling electrodes.In order to use the spatial information of EEG in the classification of motor imagery,this paper introduces a spatial representation method of EEG signals,which can help us to obtain the spatial information from the original time domain signals.The spatial information is added to DCNN to form the Third-stream Convolutional Neural Network(TCNN).We use two fusion methods :1.The full connection layers of time-domain CNN,frequeny-domain CNN and spatial CNN are connected by splicing method;2.The spatial CNN full connection layer was connected with the DCNN weight fusion layer by splicing method,and two third-stream convolutional neural networks(TCNN1 and TCNN2)were constructed.In this way,the time domain,frequency domain and spatial characteristics can be directly learned from the raw EEG data through the third-stream convolutional neural network.In this study,BCI competition dataset was used for experimental verification.The experimental results show that DCNN has better classification results than CNN using only time domain signals or frequency domain signals.The accuracy of 90.71% on BCI 2003 dataset III was higher than other results.The results on BCI 2005 dataset IIIB show that compared with previous methods such as power projection basis based feature extraction,intelligent hybrid genetic algorithm support vector machine,and phase space based EEG feature extraction,the average classification accuracy of the three subjects reaches 88.57%,and the standard deviation is the lowest.The experimental results based on BCI2008 dataset 2a show that the average accuracy of the 9 subjects reaches 74.30%,which is also better than the known methods such as channel convolutional neural network.In addition,the research method in this paper is more suitable for the classification of motor imagery than Concatenate,Add,Average and Maximum.Through experimental verification on BCI2008 dataset 2a,the average classification accuracy of TCNN1 and TCNN2 is 74.99% and 74.76%,respectively,which are superior to the time-domain and frequency-domain dual-stream convolutional neural network model(DCNN)and most of the known algorithms,proving that the spatial information of EEG can indeed help improve the performance of motor imagery recognition.Through subsequent experimental analysis,TCNN1 can also achieve a similar weight fusion effect by adjusting the number of neurons in the full connective layer output by time-domain stream,frequency-domain stream and spatial stream.The experimental conditions of each subject are different,and the best combination of the number of neurons in the full connective layer can be found for each subject through TCNN1.In this paper,a multi-stream convolutional neural network model based on time domain,frequency domain and spatial analysis is proposed,which comprehensively learns and utilizes the characteristics in time domain,frequency domain and space domain of EEG signals based on motor imagery,and provides an effective learning method for the research of brain-computer interface based on motor imagery.