Multi-modality Brain Magnetic Resonance Images Recognition With Deep Learning

The brain is the most advanced part of the human nervous system that dominates a series of vital activities of the human body.Brain-related diseases are characterized by high relapse rates,high disability rates,high morbidity rates,and high fatality rates,which challenge clinical diagnosis and treatment.Magnetic Resonance Imaging(MRI)has the advantages of multi-sequence,multi-parameter,and multi-planar imaging,which can clearly show the specific details of lesions in the patient’s brain.MRI has become one of the essential tools for clinical imaging.Different modalities of MRI data can reflect additional information about the patient’s brain from different views,how to use computers to process it has become one of the key research areas in recent years.Deep Learning(DL)technology,as a branch of Artificial Intelligence(AI),is widely used in the field of medical image processing relayed to its powerful learning ability.However,due to the complexity of disease types,medical ethics,and the high cost of data annotation,some types of diseases only have samples available with small sizes and lack lesion location annotation.In addition,the interpretability of deep learning models has also become a key problem for their application in practical medical scenarios.This paper focuses on the hot issues of deep learning models in medical image processing,including how to build efficient multitask models through parameters sharing,how to solve the classification problem of similar diseases with small available samples,and how to better explain the diagnosis process of deep models through weakly supervised localization.The first part of this paper explores how to build efficient deep learning networks with samples of small sizes.Both Neuromyelitis Optica Spectrum Disorder(NMOSD)and Multiple Sclerosis(MS)are autoimmune diseases that affect patients’ central systems.The relatively similar distribution of brain lesions and small sample size both led to the modest performance of deep learning models.In addition,the reliability of previous deep learning studies is also limited due to their invisible diagnostic process.For these reasons,a diagnostic model based on the probabilistic encoding of lesion locations(Lesion Possibility Vector Network,LPVNet)is proposed in Chapter 3 for the classification of patients with NMOSD and MS.In view of the small sample size and similar distribution of brain lesions of those two diseases,LPVNet first extracts the detailed texture features of lesions using a two-dimensional convolutional neural network and then extracts the distribution features of lesions using a fully connected neural network.The LPVNet provides a new perspective for processing threedimensional data using two-dimensional models.The experimental results show that LPVNet reduces the diagnosis error rate by 56.6% compared with the traditional 3D deep learning model,which is also superior to a senior radiologist.In addition,the special encoder-classifier structure of LPVNet allows for visualization of the inference process of the model based on the weights of the classifier,and the correctness of the visualization is further verified.The second part of this paper explores how to localize lesions in MRI images under weakly-supervised.Deep learning models have shown promising performance in various application scenarios in recent years.Researchers are not only satisfied with building high-performance deep learning models but also care about the diagnostic process of the models.However,the uninterpretable nature of deep learning has limited its practical application in medical-related tasks.In addition,medical image data face the problem of high annotation cost,thus many datasets lack the annotation of the lesions’ locations.Based on the above observation,a weakly-supervised lesion localization method(Lesion Class Activation Mapping,Lesion-CAM)is proposed in Chapter 4,which provides a visual explanation for deep learning models.Lesion-CAM achieves better lesion localization by integrating the positive gradient information in the lesion class and the negative gradient information in the no-lesion class.The experimental results show that Lesion-CAM has better lesion localization and visual interpretability than previous gradient-based visualization methods.The third part of this research paper explores how to establish the relationship between different modalities of MRI data and genotypic mutation status data.MRI techniques can be used to obtain different modality MRI data by adjusting different parameters,and this data can reflect different states of the patient’s organs from different perspectives.In addition,the mutation of specific genes has important implications for the diagnosis and treatment of diseases.Previous studies have demonstrated that patients’ imaging features can be used to predict the genotypic mutation status,but there are still problems such as single task,reliance on manual features,and predictive ability to be improved.In Chapter 5,a multi-task model incorporating multi-modality MRI images(Segmentation and Gene Prediction Network,SGPNet)is proposed for the brain lesion segmentation and prediction of IDH mutation status for glioma patients.The SGPNet uses a U-shaped structure as its backbone and applies different output modules for the lesion segmentation and IDH mutation status prediction tasks.By sharing parameters in the backbone network and extracting features from multiple levels of blocks in the expansive path,the SGPNet exhibits higher learning ability which reduces the error rate of IDH mutation prediction by 26.6% compared with previous studies.In addition,the paper further explores the impact of image features in the lesion region on IDH genotype prediction,which also increased the reliability of the model.In order to further improve the reliability of the deep learning models in this paper,the interpretability of the model is explored from multiple perspectives.In Chapter 3,the interpretability of the inference process is provided by visualizing the weights of classifiers.The visual interpretability of the deep learning model is further enriched in Chapter 4 by a weakly-supervised lesion localization method.In Chapter 5,the impact of imaging features on genotype prediction is verified by setting different training targets.The research work in this paper has strong theoretical and practical clinical application value,which provides a good working basis for building more accurate multi-modality computer-aided diagnosis models in further research work.