Research On Early Screening Of Disease Based On Multimodal Medical Imaging Data

Medical imaging is a very important diagnostic tool in the early screening,treatment and follow-up rehabilitation,monitoring and management,however there are still many problems in the course of practical treatment,such as the severe shortage of medical resources,heavy workloads on hospital staff,subjective and professional experience of the experts and so on.With the rapid development of deep learning in natural image processing and data analysis,it has recently entered also the domain of medical imaging to assist doctors in automatic diagnosis.However,considering the complexity of medical imaging recognition and the limitation of medical imaging data size,there are still some shortcomings in the mainstream of medical imaging research,because single-modal images cannot sufficiently express pathological information,while images from different modalities can provide complementary information or different visual information,make up for the shortcomings of single-modal images.Therefore,taking advantage of multimodal imaging coupled with deep learning is of great significance to assist doctors in decision-making and diagnosis.In this paper,two experiments are conducted on multimodal medical imaging data.In the first experiment,the experimental data is downloaded from Brats2020 which contains 3D brain volume images.MRI has four modalities like T1,T1c,T2 and Flair,which show different contrast and are sensitive to different brain tissues and fluid regions.The experiment used 2D brain image slices as input to mitigate overfitting,and histogram equalization technique was used to process the slices to enhance the contrast performance.Then,we developed an end-to-end architecture that uses multistream deep Convolutional Neural Network to extract and fuse features from multiple modalities respectively.Finally,we used connected the fully connected neural network for classification.Experiments using the proposed scheme have shown good results with test AUC of 95.82%,which not only is much higher than the performance of the four modalities respectively,but also has further improved the performance by 2.6%.The second experimental data in this paper comes from Shenzhen Nanshan Hospital,which has five modalities including four homologous ultrasound imaging modalities and a single heterogeneous electronic medical record modality.The task is to conduct a multi-modal early screening of Cardiovascular Disease.The hospital data is precious,but there are also many problems,the most serious problem is the lack of data.Firstly,the experiment screened the whole experimental data,and finally selected 5,060 data.Secondly,the experiment further screened and filled the ultrasound data of each modality,in the meantime,processed the structured data and then associated the personal unique identification of electronic medical record to merge into the new electronic medical record,then we used the pre-trained multi-channel convolutional neural network to extract and combines the features of four ultrasound modalities.And the extracted features were merged with the new electronic medical record through personal unique identification.Finally,the data was used for training in the model constructed with the XGBoost.Experiments using the heterogeneous multimodal scheme shows good results with test AUC of 85.75%,which is higher than the homologous multimodal scheme(with test AUC of 83.39%)and the single modality scheme respectively.It is not easy to obtain the hospital data,for the reason that there are many restrictions such as procedure,privacy security and so on,but the experiment with the hospital data can reflect the performance of deep learning in clinical application,it has strong theoretical significance and practical reference.The paper reflects that multimodal medical imaging data,even multimodal medical data has great potential for data mining.Deep learning based on multimodal medical imaging data can mine different pathological information from different modal medical data and provide more complementary information.It can describe the disease more efficiently,comprehensively and accurately,and assist doctors in decision-making and diagnosis,which will lay a solid foundation for the construction of big data platform of intelligent medicine in the future.