Research On Multimodal Fusion For The Early Alzheimer’s Disease Identification

Mild cognitive impairment is the early state of Alzheimer’s disease(AD).Currently,it is generally believed that mild cognitive impairment is a key link in the screening and treatment of AD,and can effectively delay the development of AD.Therefore,the early Alzheimer’s disease identification is crucial.The rapid development of medical informatization,neuroimaging,gene sequencing and other technologies has provided rich multimodal data for the early Alzheimer’s disease identification.Compared with multimodal data,single modal data often has limitations in information expression,and multimodal fusion is still a challenging problem.To this end,this thesis carries out research on multimodal fusion for early Alzheimer’s disease identification with magnetic resonance images(MR images),clinical information and genomic data as data carriers and deep learning as technical background.The main research work and innovations are shown as follows:(1)A multimodal multiscale convolutional neural network for the early Alzheimer’s disease identification is proposed.Clinical information and MR images provide different information for the early Alzheimer’s disease identification,and their fusion may help improve the identification performance.Therefore,a multimodal convolutional neural network for the early Alzheimer’s disease identification is proposed.Considering that representation learning ability of the single-scale may be insufficient,this thesis proposes a multiscale representation for MR images feature learning.Considering that different clinical information has different properties,for example,clinical scale scoring requires the cooperation of the subjects,which is subjective to a certain extent,this thesis designs a multimodal joint learning strategy: objective clinical information is fused with features of MR images;and the subjective clinical scale score is constructed as an auxiliary task to conduct multi-task learning with identification.Experimental results show that the proposed method has superior performance in the early Alzheimer’s disease identification.(2)A multimodal adaptive graph convolutional network for the early Alzheimer’s disease identification is proposed.AD has been found to be highly correlated with genetics.In order to include single nucleotide polymorphism(SNP)data in MR images and clinical information,a multimodal adaptive graph convolutional network for the early Alzheimer’s disease identification is proposed.In this thesis,a cross-modal feature fusion module is designed to fuse MR images and SNP features by cross-modal attention.Associations among Subject may be useful for identification,using clinical information to measure associations,and constructing population graph that combine clinical information with other modal.However,the population graph constructed by handcrafting is static and unlearned,and may not be well combined with graph convolution network.To solve this problem,an adaptive graph convolutional network is proposed from the view of graph structure learning.Experimental results show that this method can further improve the performance of the early Alzheimer’s disease identification,and the multimodal fusion is more conducive to the early Alzheimer’s disease identification than the singlemodal.Moreover,some important brain regions associated with the early Alzheimer’s disease identification are identified.