Multimodal Learning With Application To Medical Data Analysis

Collected data are usually heterogeneous and from multiple sources.More and more practical problems are involved with data of multiple views.Typically in medical scenarios,doctors need to analyze the condition for patients in a variety of ways before giving a final diagnosis.For some diseases which is hard to diagnose,it is impossible to analyze in only one way.Therefore,from the clinical perspective,medical analysis with multimodal data ensembled is much more reasonable and accurate.The deep convolutional neural networks have been proved to perform well in diverse medical images analysis.However,this conflicts with the multimodal learning in some degree.Specially,a well-trained deep model can’t generalize well on test data that are from different distribution,which is a major challenge.Requiring more training data for data and tasks are so far the best solution under the circumstance of supervised learning,though,delineations are usually at the cost of time and efforts,which cause high expense in practice.Thus,an unsupervised domain adaptation for crossmodal segmentation with multiple constraints is proposed,which aims at extracting domain-invariant feature by learning similarity between different modality.Without labels in target domain,it transfers knowledge from source to target for medical image segmentation to reduce the demand for delineations and improve the generalization of the model.Especially,the model is constrained to be domain-invariant in three levels.Generative Adversarial Networks is applied for image-level domain mapping to use the transformed data as training data.In the meanwhile,an encoder-shared generator and encoder-decoder based image reconstruction network are designed to constrain the extracted features to be common between modalities.Semantic space is alignedby adversarial training with extra discriminator and domain confusion constrain.With complementary constraints ensembled in one framework,proposed method is evaluated on cardiac segmentation in the setting of unsupervised domain adaptation and achieved a better result than existed methods.Ablation study is conducted to analyse the benefit of each constrain.Afterwards,a multimodal learning based model called Prescription vi A Learning l Atent Symptoms(PALAS)is proposed to answer “could the computer automatically predict the suitable drugs for Parkinson’s Disease(PD)patients”,and provides datadriven reference for neuro-pathologists.Specifically,with recorded motor and nonmotor symptoms observed,a multi-modality representation is designed by integrating both the feature-based and similarity-based representations.Then,PALAS learns a latent symptom space to better model the relationship between the observed symptoms and the prescription drug,as there is a large semantic gap between them according to observation.Moreover,an efficient alternating optimization method is proposed for PALAS.Experimental results on data collected from 136 PD patients demonstrate the effectiveness.Compared with existing methods,PALAS can predict proper prescription for PD patients and shows great clinical potential.