The Study On Multimodal Neurobiological Data Analysis And Brain Disease Recognition Based On Deep Learning

Multimodal neurobiological data analysis is an important research direction in the interdisciplinary study of brain diseases and artificial intelligence.Clinical studies usually collect a large number of multimodal neurobiological information such as neuroimaging,electrophysiological and behavioral data from patients,which have provided vital supports for brain diseases from diagnosis to treatment.Deep learning provides effective technical supports for exploring and understanding the clinical differences and pathogenesis of brain diseases.It is of great importance and pratical value to develop accurate and efficient techniques for brain disease recognition based on deep learning and multimodal neurobiological data,and to explore neurobiomarkers for brain diseases.Therefore,based on multimodal neurobiological data,including attention behavior data,diffusion tensor magnetic resonance imaging data and EEG data,a variety of disease-specific deep learning models are proposed in this dissertation,and the accurate recognition and differentiation of various brain diseases and brain signals are realized.Based on the proposed explainable artificial intelligence method,the biological visual features of brain diseases and the brain-like neural mechanism in the brain network are discovered.The research work of this dissertation provides new methods for clinical assistant diagnosis of brain diseases and motor function-assisted rehabilitation.This dessertation mainly includes three parts of the research work.Study 1 analyzed the visual behavior of autism based on eye movement data and two-stream ASD-VGG network.The visual features associated with atypical attention in autism spectrum disorder（ASD）,especially those that can accurately discerning between ASD and typical development（TD）at the individual level,largely remains unknown.There is also a lack of explainable artificial intelligence（XAI）-based analysis that is able to reveal,visualize,and explain critical features underlying classification.In this study,a novel XAI deep learning framework is developed for systematic identification and in-depth exploring of critical visual features that can accurately discerning between ASD and TD at the individual level.The framework integrates deep learning classification model,image segmentation method,image ablation method and a direct measurement of classification ability（AUC）.First,we designed a new twostream ASD-VGG network,which fused the stimulus information of 700 natural scene images and the corresponding eye movement information into the two-stream network.The accuracy of ASD recognition was 0.95,which is more than 3% higher compared with other SOTA（state-of-the-art）models.Based on the XAI framework,we systematically studied the classification ability of visual features at three different levels:single-image level,multi-image level and local-feature level.We found that the contribution of visual features at the single-image level was very limited,while the AUC value based on the combination information of the top-250 images（0.96）exceeded that of 700 images（0.93）,indicating that there were both synergistic and antagonistic effects during image combination.We excavated 11 kinds of local visual features from natural scene images that played an important role in ASD recognition,with two new categories of discriminative visual features（Food-drink and Outdoorobjects）,and two improved categories of discriminative visual features（Center-object and Animals）.And we also found a group of the top-9 feature combinations with a regional area of only 44%,and the AUC reached 0.92.The efficient feature combination is helpful to guide and train doctors in clinical assistant diagnosis.Finally,a data subset containing only 12 images with an AUC of 0.86 was obtained by recursive feature elimination method.This efficient data subset can be applied to help families to screen independently by the phone app.In summary,our XAI deep learning framework provides a novel and powerful tool for identifying and understanding atypical visual attention in ASD,and will in turn facilitate the identification of autism biomarkers.Study 2 identified a variety of brain diseases based on diffusion tensor magnetic resonance imaging data and Res-GCN network.The heterogeneity and comorbid symptoms within brain diseases has become a great challenge for classification multiple brain diseases in clinical practice,while there is still a lack of general deep learning framework that could be effectively applied to different detection tasks of brain diseases,especially the lack of distinction between various brain diseases.In this study,by combining the population type of graph convolution network（GCN）and the shortcut connection structure in residual neural network（ResNet）,a new general deep learning framework Res-GCN was developed.Based on diffusion tensor magnetic resonance imaging（DTI）data,three brain diseases such as schizophrenia（SZ）,major depression disorder（MDD）and bipolar disorder（BD）were detected respectively,and the distinction between these three brain diseases was realized.First,we trained and tested the DTI data of SZ,MDD and BD based on the constructed 3D-ResNet21 network,and the accuracy was 0.8050,0.7011 and 0.7332,respectively,which verified the effectiveness of the pre-training network C3D-UCF101 Net feature extraction.Combining the DTI features extracted by C3D-UCF101 Net and other medical information,and based on GCN network,we designed a novel Res-GCN network to improve the detection accuracy of three brain diseases.The accuracy of SZ,MDD and BD was 0.8243,0.9329 and 0.8963,respectively.The accuracy of MDD is more than8% higher compared with other SOTA models,and the accuracy of BD is more than 1%higher compared with other SOTA models.The high precision of different recognition tasks verifies the generalization ability and robustness of Res-GCN.Finally,we migrated the Res-GCN network to the four-class classification tasks of three brain diseases and healthy people,and the classification accuracy was 0.6215.In summary,our Res-GCN deep learning network provides a novel and effective general tool for clinical diagnosis of brain diseases,and provides a new idea for diagnosis and treatment of multiple brain diseases.Study 3 distinguished motor-imagery behaviors based on EEG data and multiscale CNN-Trans network.The accurate classification of motor-imagery EEG signals in brain-computer interface（BCI）system is challenging.Deep learning has become a powerful tool to study motor-imagery behavior,but there is still a lack of end-to-end deep learning framework combining neural mechanism,which can effectively solve the long-range dependence of EEG data and accurately classify motor-imagery EEG,especially the lack of a general model for classifying motor-imagery signal using crossindividual transfer.Based on multi-head self-attention mechanism,we developed a novel multi-scale CNN-Trans deep learning model for three classification tasks of motor-imagery EEG: two-class of left/right hand motor-imagery（L/R）,three-class of left/right hand motor-imagery and opening eyes（L/R/O）,and four-class of left/right hand motor-imagery and opening eyes and feet motor-imagery（L/R/O/F）.Based on Transformer and CNN structure,a CNN-Transformer module was designed.Multiscale CNN-Trans deep learning model can extract features of different time scales by integrating multiple CNN-Transformer modules.First,we adopted the Physionet EEG motor-imagery public dataset to train and test the multi-scale CNN-Trans model.Based on the sample with 3-second and the cross-individual training,the test accuracy of L/R two-class classification,L/R/O three-class classification and L/R/O/F four-class classification was 82.68%,73.76% and 63.24%,respectively.The accuracy of two-class classification is more thran 0.2% higher compared with other SOTA models,the accuracy of three-class classification is more than 1.4% compared with other SOTA models,and the accuracy of four-class classification is more than 0.1% higher compared with other SOTA models.Then,we explored the influence of different scale numbers on the model performance.When the scale number was 4,the model had the best comprehensive performance,and the increase of scale number could improve the classification ability to some extent,especially in L/R/O three-class classification task.Finally,by analyzing the brain topography of attention weights,we found that the attention weights corresponding to the sensorimotor areas revealed a pattern of eventrelated desynchronization（ERD）which was consistent with μ rhythms and β wave.The visualization result was helpful to understand the working mechanism of the model in EEG classification.Our multi-scale CNN-Trans model provides a novel and promising method for understanding and classification motor-imagery behaviors,which plays a positive role in promoting the development of BCI systems.In this dessertation,by analyzing multimodal neurobiological data,three diseasespecific deep learning models were proposed to detect and identify a variety of brain diseases and brain signals,and some critical biological visual features and brain-like neural mechanisms were found.We have finished some basic research work for the construction of multimodal and multitask assistant diagnosis system for brain diseases.Due to the limitations of data-driven,there are still some shortcomings in the construction of this system: the lack of joint use of multimodal data,and the lack of exploration of multitask deep learning general frameworks for multiple brain disease identification tasks.In order to further improve the system,based on the research basis of this dessertation,in the future,we will explore in depth from three aspects:combining multimodal data such as neural data,behavioral data,genetic data and clinical diagnosis information to obtain a higher level of classification,developing a multitask assistant diagnosis system for large-scale screening of brain diseases,and exploring common neurobiological markers for brain diseases in neuroimaging and behavioral dimensions.