Research On Deep Transfer Learning Methods For Motor Imagery Electroencephalogram

Stroke threatens the human health and life safety seriously.Especially in China,the annual stroke incidence is significantly higher than that of overall the world,the number of stroke survivors could be as high as 13 million,and about three-quarters suffer from limb movement disorders,which brings heavy burden to family and country.The China Brain Project was released in 2016 to improve the diagnosis and treatment of major brain diseases,and the multidisciplinary fusion of brain science and artificial intelligence etc.is further worked into the 14th Five-Year Plan as the national major science and technology strategy in 2021.Therefore,it has an important theoretical significance and application prospect to study intelligent and efficient motor nerve rehabilitation methods.Brain-computer interface technology based on motor imagery electroencephalography（MI-EEG）has demonstrated its specific advancement,effectiveness and brain-computer interaction in motor neurorehabilitation,and the accurate recognition of MI-EEG is the key.Recognizing MI-EEG by using deep learning technology can avoid or reduce the influence of feature engineering and human factors,but it faces the dilemma of small data amount and insufficient model training.Combining with transfer learning makes it possible to borrow data or models.However,each person has his/her own unique brain anatomy structure,function and neural activity pattern,this results in the obvious individual variability of MI-EEG,which also changes with MI tasks and time,thus the adaptability and generalization ability of deep transfer learning methods are weakened.In this thesis,based on data alignment,domain adaptation,knowledge distillation and optimization techniques,an in-depth study of deep transfer learning methods of MI-EEG will be carried out from single source domain to multi-source domains and in multiple scenarios,such as across subjects,tasks and even datasets,enhancing the transfer effect of deep neural network models.The main research results are as follows:（1）A correlation distance-based source domain optimization and block parameter transfer methodTo improve the effect of single source domain parameter transfer,a correlation distance-based source domain optimization and block parameter transfer method is proposed.First,Pearson correlation distance is used to optimize the source domain data,and the short-time Fourier transform is performed to acquire the time-frequency spectrogram images;then,a simplified shallow VGG-16 convolutional neural network（CNN）is designed and the time-frequency spectrogram images of source domain are employed to train model;furthermore,block-based frozen-fine-tuning transfer strategy is used to accomplish the parameter transfer and fine-tuning of model.Cross-dataset transfer learning experiments are conducted based on two public MI-EEG datasets,the average recognition accuracy and Kappa value of 9 subjects with 10-fold cross validation are 94.79%and 0.8980,respectively.The results show that the high-quality source domain with larger inter-class distance is beneficial for enhancing source model performance,and the block-based frozen-fine-tuning transfer strategy is able to quickly and automatically find and freeze the block with the greatest contribution in the source model,improving the transfer efficiency and effectively reducing the training time of the target model.（2）A dual alignment-based multi-source domain adaptation methodTo reduce the influence of domain shift in multi-source transfer learning,a dual alignment-based multi-source domain adaptation method（DAMSDA）is proposed.First,Time-frequency analysis of multi-channel of MI-EEG signals are performed based on continuous wavelet transform,and the time-frequency spectrogram images corresponding to the frequency band 0-32 Hz are stitched to construct multi-source domains and target domain;then,the pre-trained Res Net50 network is applied to find the decision boundary of the target domain,and the normalized mutual information between each source domain sample and the target domain information sample is computed for source sample weighted alignment;next,a maximum mean discrepancy（MMD）loss embedded multi-branch deep network（MBDN）is designed to align the specific feature distribution of each pair of source and target domains in the reproducing kernel Hilbert space;finally,the recognition accuracies of each single-branch network are ranked in descending order and used for weighted prediction of MBDN to automatically determine the optimal number of source domains.Transfer learning experiments of the same MI tasks are conducted based on three public MI-EEG datasets,the 2-class and 4-class recognition accuracies of cross-subject transfer are 92.56%and 69.45%,respectively,and the recognition accuracy of cross-dataset transfer is 89.57%.The results show that the sample and feature alignments greatly enhance the same MI tasks feature distribution similarity of each pair of source and target domains,and the weighted prediction of MBDN can adaptively match part of the source domains to further improve the ability of coping with domain shift.The superiority of DAMSDAis displayed with the Kappa value and t-test as well as other statistical analysis.（3）A partial domain adaptation method based on central alignment and weighted adversarial learningTo mitigate the negative transfer of outlier source classes in partial transfer learning,a partial domain adaptation method based on central alignment and weighted adversarial learning is proposed.First,the signal augmentation and feature enhancement of per-channel MI-EEG are performed based on wavelet packet transform;then,all the source and target domains are respectively executed centroid alignment based on Riemannian manifolds;next,a weighted selective adversarial network（WSAN）is designed,and the“soft labels”generated from the target domain are simultaneously used for class-level weighting of the MMD,the domain discriminator bank and classifier loss functions;finally,the loss parameters of the network are optimized based on the particle swarm optimization algorithm.Partial transfer learning experiments are conducted with two public MI-EEG datasets,the average recognition accuracies of cross-subject transfer from four classes to two classes,three classes and three classes to two classes are 79.64%,73.58%and 84.32%,respectively;the average recognition accuracies of cross-dataset transfer from four classes to two classes and three classes to two classes are 87.25%and 91.87%,respectively.The results show that after reducing subject differences via Riemannian center alignment,the selective adversarial learning and loss parameter optimization of WSAN further improve the consistency of the same classes feature distributions in the source and target domains,achieving the enhancement of the same classes positive transfer and the attenuation of the outlier source classes negative transfer,and pass the statistical test.（4）A cross-class transfer learning method with two-level alignment and knowledge distillationTo address the cross-class data distribution variability between multi-source and target domains,a cross-class transfer learning method（CTL）with two-level alignment and knowledge distillation is proposed.First,the center samples of each class in the source and target domains are separately obtained by using fast partitioning around medoids algorithm,and the center alignment matrix is constructed to perform the1^st-level domain alignment for anyone possible class to class transitive correspondence（CCTC）;then,the 2^nd-level subject alignment is performed by Euclidean alignment for each class of all source domain.Furthermore,the central samples of target domain together with the two-level aligned source domain are mapped into tangent space,the resulting features successively train the teacher CNN and its corresponding student CNN（SCNN）after parameter transfer,and the parameters of distillation loss are optimized automatically by a scaling-based grid search method.Cross-class transfer learning experiments are conducted on a public MI-EEG dataset with multiple MI tasks,the 2-class and 4-class average recognition accuracies of simultaneous cross-class and cross-subject transfer are 86.37%and72.48%,respectively.The results show that two-level alignment is helpful for increasing the data distribution similarity across classes,and the knowledge distillation embedded and parameter-optimized CTL can automatically find the optimal SCNN associated with the best CCTC,which increases the adaptability of the method.The optimal SCNN combines great inheritance and learning capabilities,effectively improving the effect of cross-class transfer learning,and demonstrating the statistical performance advantage.（5）Deep Transfer Learning-based Upper Limb Grasping Motor Training Game DesignTo verify the effectiveness and feasibility of the proposed transfer learning methods in motor training scenarios,a motor training game is designed for upper limb grasping ability based on deep transfer learning method and Unity 3D.First,an immersive virtual environment based on Unity 3D was created to simulate the action of grabbing a cup in daily life;then,model building,pre-training and parameter/model transferring are performed by using the public dataset and the self-harvested data containing four MI classes based on the multiple transfer learning methods proposed in Chapters 2 to 5 to obtain the target domain model;next,the prediction results of the target domain model are applied to drive the left and right movement,grasping or internal rotation actions of the virtual arm,while updating the prediction score and correct rate in real time;finally,whether the game will pass and enter more classes of motor training is determined according to whether the transfer model predicted correct rate exceeds 80.00%.The results show that the proposed transfer learning methods better accomplish the parameter/model transfer across subjects,classes and even datasets,and can be applied to practical motor training,verifying the adaptability and application value of them in this paper.The research findings of this paper have a positive promotion and reference role for the development and application of BCI technology in the field of neurorehabilitation.