Multimodal Breast Cancer Diagnosis Based On Multi-Level Fusion Network

According to data published in 2020 by the International Agency for Research on Cancer(IARC),a division of the World Health Organization(WHO),breast cancer is the most serious disease causing death in women,and men are also at risk of developing it,and its incidence is increasing year by year with a younger patient population.Early qualitative diagnosis of breast cancer and the development of protocols for timely treatment can significantly reduce the mortality rate of breast cancer.Traditional breast cancer diagnosis mainly consists of histopathological examination and physician consultation,however,these modalities not only require specialized domain knowledge,but also the histological structure of pathological images is very complex and there is no uniform standard,and their results are highly susceptible to subjective artificial factors by medical experts.(1)In response to the existing breast cancer data with single features and poor generalization ability,the paper proposes a multimodal data feature extraction algorithm based on multilayer fusion network.The model consists of three main modules:firstly,a DenseNetGAT model is proposed to extract pathology image features,which models the structural continuity and interactions between each pathology image of a patient and preserves the consistent or compensatory lesion information existing between images.Secondly,the structured electronic medical records are subjected to data dimensionality reduction using principal component analysis(PCA),and then feature extraction is performed using multilayer perceptron(MLP)to reduce the redundancy of data information and reduce the complexity of the model.Finally,using the pre-trained language model ClinicalBERT to extract the textual features of patients’ medical records can effectively avoid gradient explosion and loss.It is proved through experiments that the proposed model in this paper provides a richer feature representation.(2)For the heterogeneity within the data when fusing multimodal data,the paper proposes a multimodal data feature fusion method with Multimodal Attention Gate(MIG).The core idea is to adjust the representation of one modality with the displacement vectors obtained from other modalities,and the modality being adjusted is called the primary modality and the other modalities are called the secondary modalities.The experimental results demonstrate that multimodal self-attentive gates can be used to fully exploit the complementarity among the modal data and lead to more comprehensive feature representations and more accurate disease prediction and diagnosis performance,Its diagnostic accuracy reached 93.62%,which is better than some existing models.(3)A breast cancer diagnosis system based on a multilayer fusion network was designed and developed,which integrates the above algorithms with clinical needs to assist clinicians in diagnosing breast cancer,increasing the percentage of correct decisions and avoiding unnecessary surgical procedures.It enables physicians to develop appropriate treatment plans in a timely manner,greatly improving diagnostic accuracy as well as efficiency.