Research On Breast Tumor Classification Method Based On Multi-modality Fusion

Breast cancer is a serious public health problem with a rapidly increasing incidence among women.The International Agency for Research on Cancer has released the latest global cancer data in 2020,showing that breast cancer has displaced lung cancer as the most common cancer,ranking first in incidence and fifth in mortality,respectively.More than90% of patients with early-stage breast cancer can be cured,while the cure rate for patients with middle-stage and terminal-stage breast cancer has decreased from 70% to 50%.Therefore,early detection and treatment play an important role in preventing the spread of breast cancer.In clinic,medical imaging is an important auxiliary means for the early detection of breast cancer.Magnetic resonance imaging(MRI)is widely used in the diagnosis of breast cancer due to its higher spatial and temporal resolution,which is lacking of radiation damage and able to explore more subtle lesion areas.With the development of medical image acquisition system,the single limitation of single modality image is particularly prominent,and the information complementarity between the multi-modality image breaks the inherent limitation of single modality,so the multi-modality image fusion technology has been developed rapidly.MRI includes diffusion weighted imaging technology and apparent diffusion coefficient scan sequences,and there has correlation and complementarity between DWI and ADC.Therefore,this paper proposes a series of multi-modality fusion methods based on DWI and ADC to classify breast tumors.The main contributions are as follows:(1)This paper proposes a classification method for breast tumors based on multi-modality correlation attention.To solve the problem that the existing multi-modality fusion methods fail to make full use of the multi-modality correlation information,this paper develops an attention network based on the consistency regularization term,and constructs a multi-modal correlation attention module to explore the correlation information between the two modalities and makes the network pay more attention to the features of the region with high correlation.The classification consistency module is used to minimize the label differences between DWI and ADC modalities,which ensures the classification consistency of different modalities in the same patients,thus improving the recognition ability of single-modality images and the accuracy of multi-modality classification of breast tumors.Experimental results demonstrate that the multi-modality relation attention method outperforms existing multi-modality fusion methods.The area under the receiver operating characteristic curve,accuracy,specificity,and sensitivity are 85.1%,86.7%,83.3%,and88.9% respectively.(2)This paper proposes a classification method of breast tumors based on triple inter-modality interaction.In view of the problem that the existing multi-modality fusion methods ignore the interaction between modalities and fail to make full use of multi-modality information,resulting in the weak differentiation of the single modality features.This paper constructs a triple interaction mechanism between modalities by introducing a correlation interaction module firstly,which utilizes the prior knowledge of the correlation between the modalities of clinical DWI and ADC.Meanwhile,the attention mechanism is applied to enhance the representation of the single modality information.Next,a channel interaction module is proposed,which uses dense connection in channel dimension to fuse the information of two modalities,enhancing the interaction between modalities.Immediately after,the discriminant interaction module is introduced,and the attention mechanism is used to obtain the more discriminant single-modality features,which effectively guides the network to focus on the more critical areas for the classification task.All modules are embedded in a joint framework.Through these interactions between the three modules,the ability of single modality feature representation is enhanced,and the performance of the multi-modality classification system of breast tumors is improved ultimately.The experimental results demonstrate that the interaction between modalities is beneficial for improving the accuracy of multi-modality classification tasks.Extensive ablation studies have been carried out,which provably affirms the advantages of each component.The area under the receiver operating characteristic curve,accuracy,specificity,and sensitivity are 87.0%,87.0%,88.0%,and 86.0%,respectively.(3)This paper presents a classification method of breast tumors based on inter-and intra-modality interaction.In view of the problem that existing multi-modality fusion methods only consider one interaction within or between modalities,which fails to fully explore the discriminative information.A network combining inter-modality interaction and intra-modality interaction is proposed for the classification of breast tumors.In order to make full use of multi-modality correlation,complementary,and discriminative information,three interaction mechanisms between modalities are introduced,including inter-modal relationship interaction,channel interaction,and multi-level attention fusion interaction.The intra-modality interaction mechanism is introduced and a new dual-parallel attention module is used to further improve the discrimination ability of single-modality features.At the same time,combined with the multiple interactions within and between modalities,the multi-modality information is fully integrated,and the performance of multi-modality breast tumor classification is improved.Experimental results demonstrate that the proposed method outperforms other multi-modality fusion methods,and extensive ablation studies are conducted to verify the advantages of each interaction module.The area under the receiver operating characteristic curve,accuracy,specificity,and sensitivity are 90.5%,89.0%,85.6%,and 92.4%,respectively.