Hierarchical Multi-modality Fusion Strategy And Its Application In Radiomics

Radiomics is a technique used to extract high-throughput quanti-tative image features to create mineable data from radiological images to provide valuable diagnostic,prognostic,or predictive information in a clinical decision making context.At present,a large number of studies have proved the advantages of radiomics applied on organ toxicity prediction,survival analysis,classification of tumor and et.al.However,from a practical point of view,challenges still exist in the management of different sources of f’eatures,such as handcrafted radiological features,clinical features,and automatic learned features,as well as in terms of preference of a proper classification model to accomplish a particular prediction task.In this paer,we proposed a novel multiple classifier fusion method for the classifier selection problem,which aviods the process of selecting the "best" classifier.And a hierarchical fusion framework based on the proposed method is also investigated to reliably link the multi-modality features,which proposes a new way of the manner of combining the miscellaneous information.1)Multi-criterion decision-making based(MCDM-based)classifier fusion strategy(MCF).In this study,the classifier fusion is,in essence,a multi-criterion decision-making problem to deal with situations where a set of alternatives(which represent multiple classfiers)exists,which try to assign a weight to each alternative in the set of alternatives by evaluating certain evaluation indicators or criteria.Ten public UCI datasets for binary classification were tested to assess the merit of the proposed fusion method in classifier fusion with unitary modality feature input.For all datasets,the proposed method generally performed better than other five benchmark methods in terms of accuracy,AUC,sensitivity and specificity.Specifically,statistical significant improvements in accuracy and AUC were obtained by statistical analysising based on the datasets.2)Hierarchical fusion framework for multi-modality and multiple classifiers.In this study,a hierarchical multi-modality and multi-classifier predictive fusion model was built upon the proposed MCDM-based classifier fusion method.Classifiers and modalities(which was also regarded as a set of alternatives)were respectively fused via the proposed MCF at the first and second level.The framework was evaluated on two clinical datasets.The first one is the survival prediction in non-small cell lung cancer,with fusing clinical parameters and radiomics features with five base classifiers,the framework achieved the performance on accuracy,AUC,sensitivity and specificity were respectively 0.75,0.81,0.73 and 0.80,which is the best performance than other fusion schemes.The senond one is the rectal toxicity prediction in prostate cancer radiotherapy.By fusing clinical parameters and dosimetric features,the framework achieved the better performance than other fusion schemes in terms of accuracy(0.78),AUC(0.83),sensitivity(0.76)and specificity(0.79).3)The multi-modality and multi-channel fusion in breast cancer discrimination.This study is based on the previous studies to modify the structure of the hierarchical fusion framework to adapt to the study of the classification of breast tissue into normal,benign and maligant categories,and extend the classifier fusion method from binary classification to mutiple classification.Combining the handcrafted features and automatically learned features from deep convolutional neural networks(DCNN)both on digital breast tomosynthesis(DBT)and full-field digital mammography(FFDM)to establish a more accurate breast tissue prediction model,with performance on mattews correlation coefficient(MCC)is 81.72%.In this study,we proposed a multiple classifier fusion model and a hierarchical fusion framework for multi-modality features,and validated on ten public UCI datasets and three clinical datasets respectively.The Experimental results show that the algorithm and framework are feasible and effective.