Multi-Modal Fusion And Multi-Center Harmonization Methods In Medical Image For Diagnosis And Prognosis

Medical imaging plays a crucial role in clinical decision-making,as different imaging modalities provide unique information about various aspects of a disease.Integrating this information effectively can help doctors more accurately assess a patients condition and develop personalized treatment plans.However,traditional multimodal fusion approaches often suffer from information redundancy and limited interaction between modalities,limiting their performance.Furthermore,variations in scanning equipment and acquisition parameters across different centers often lead to batch effects between images,hindering the generalization of diagnostic and prognostic models.Therefore,developing novel and effective multimodal fusion methods and conducting multi-center medical image harmonization is critical for successful implementation of such models in large-scale clinical practice.This thesis focuses on addressing these challenges through two main approaches:multimodal fusion and multi-center harmonization.(1)BAF-Net:Bidirectional Attention-aware Fluid Pyramid Feature Integrated Multi-modal Fusion Network for Diagnosis and PrognosisTo go beyond the inherent deficiencies of current three prevalent multi-modal fusion strategies(i.e.,pixel-,feature-and decision-level fusion),this work proposes a bidirectional attention-aware fluid pyramid features integrated fusion network(BAFNet)with cross-modal interactions for multi-modality medical images diagnosis and prognosis.The network is composed of two identical branches to preserve the unimodal feature and one paralleled bidirectional attention-aware distillation stream to assimilate cross-modal complements progressively and learn supplementary refined features in both the bottom-up and the top-down processes.The fluid pyramid connections were adopted to integrate the hierarchical features at different levels of deep neural network,and the depth-wise separable convolution was introduced to fuse the cross-modal crosslevel features to alleviate the increase of parameters to a great extent.Extensive experiments on two public datasets and one in-house dataset demonstrate higher performance and superior robustness to the scale of dataset than three conventional fusion strategies and unimodal network in diagnosis and prognosis.(2)Optimal Batch Determination for Improved Harmonization and Prognostication of Multi-center PET/CT Radiomics Feature in Head and Neck CancerIn order to improve the generalization performance of head and neck cancer radiomics prognostic models in multi-center applications,two batch effect correction methods(including unsupervised harmonization and supervised harmonization)were studied in this work.Unsupervised harmonization identified the batch labels by Kmeans clustering for Combat.Supervised harmonization applied ComBat separately or sequentially to correct given batch effects(center effect,manufacturer effect and scanning device effect,etc.).A total of 800 PET/CT images of head and neck cancer patients from 9 centers were collected in the experiment,and a Cox model wasconstructed to predict the patients overall survival(OS).Results show that compared to original models without harmonization,Combat harmonization would be beneficial in OS prediction with concordance index(C-index)of 0.687-0.740 vs.0.684-0.767 in external validation cohort.Models with supervised harmonization slightly outperformed models with unsupervised harmonization(C-index:0.692-0.767 vs.0.684-0.750 for external validation cohort).Sequential harmonization further improved the performance with the highest C-index of 0.767,which determines the order of the batch labels followed by the reconstruction algorithm effect and then the scanner effect.Therefore,sequential harmonization after clarifying the optimal batch label order has great potential to improve the performance of radiomics model prognosis.