Application Of Multimodal Image Data Analysis Based On Deep Learning In Cervical Cancer Screening

Purpose: The primary aim of this study was to evaluate the diagnostic accuracy of colposcopy in identifying high-grade squamous intraepithelial lesion or worse(HSIL+)and the characteristic performance of colposcopic images with various severity levels of cervical lesions.Materials and Methods: This study collected 1964 female cases who underwent colposcopy at Tongji University Affiliated Hospital from February 2016 to March 2019.The collected materials included human papillomavirus(HPV)test results,cervical cytology test results,colposcopy images and Histopathological biopsy results under the guidance of colposcopy.All colposcopy images were obtained from the standard-of-care colposcope(Leisegang 3ML LED)and evaluated by two colposcopy specialists(senior colposcopy specialist and general colposcopy specialist)according to the 2011 International Federation of Cervical Pathology and Colposcopy Colposcopy(IFCPC)Colposcopy Standards.Using histopathological biopsy results as the gold standard,the diagnostic performance of different colposcopy specialists and the colposcopy image characteristics of cervical high-grade squamous intraepithelial lesions or worse lesions were analyzed.The diagnostic performance of the test model in the independent test set was assessed using the area under the curve(AUC)of the receiver operating curve.Statistical analysis was performed using linear association,Pearson’s χ test,Kappa test,and risk test.Results: The consistency between the diagnosis of ordinary colposcopy experts and biopsy pathology was 59.22%,and the kappa coefficient was 0.465;the consistency between the diagnosis and biopsy pathology of senior colposcopy experts was 67.26%,and the kappa coefficient was 0.569,which were higher than those of ordinary colposcopy.expert.The sensitivity,specificity and AUC of common colposcopy experts,senior colposcopy experts and cytology examination for HSIL+ were 57.14%,93.82%,0.76;71.57%,94.05%,0.83 and 26.69%,98.72%,0.63,respectively.The concordance between the assessment results and pathology of senior colposcopy experts was higher than that of ordinary colposcopy experts,and colposcopy diagnosis was more likely to underestimate cervical lesions than biopsy pathology.The colposcopic features of HSIL+ were as follows:(1)thick or bulgy acetowhite epithelium with sharp border;(2)completely nonstained of Lugol’s iodine;(3)type III/IV/V of gland openings;(4)punctation or atypical vessels.Conclusion: Colposcopy images of HSIL+ lesions have certain characteristics.Although the diagnostic accuracy of senior colposcopy experts is higher,colposcopy diagnosis is more likely to underestimate cervical lesions,suggesting that the accuracy of colposcopy needs to be further improved.Purpose: To develop and evaluate the colposcopy based deep learning model using all kinds of cervical images for cervical screening,and investigate the synergetic benefits of the colposcopy,the cytology test,and the HPV test for improving cervical screening performance.Materials and Methods: This study consisted of 2160 women who underwent cervical screening,there were 442 cases with the histopathological confirmed high-grade squamous intraepithelial lesion(HSIL)or cancer,and the remained 1718 women were controls.Three kinds of cervical images were acquired from colposcopy including the saline image of cervix after saline irrigation,the acetic acid image of cervix after applying acetic acid solution,and the iodine image of cervix after applying Lugol’s iodine solution.Each kind of image was used to build a single-image based deep learning model by the VGG-16 convolutional neural network,respectively.A multiple-images based deep learning model was built using multivariable logistic regression(MLR)by combining the single-image based models.The performance of the visual inspection was also obtained.The results of the cytology test and HPV test were used to build a Cytology-HPV joint diagnostic model by MLR.Finally,a cross-modal integrated model was built using MLR by combining the multiple-images based deep learning model,the cytology test results,and the HPV test results.The performances of models were tested in an independent test set using the area under the receiver operating characteristic curve(AUC).Results: The saline image,acetic acid image,and iodine image based deep learning models had AUC of 0.760,0.791,and 0.840.The multiple-images based deep learning model achieved an improved AUC of 0.845.The AUC of the visual inspection was 0.751.The Cytology-HPV joint diagnostic model had an AUC of 0.837,which was higher than the cytology test(AUC = 0.749)and the HPV test(AUC = 0.742).The cross-modal integrated model achieved the best performance with AUC of 0.921.Conclusions: Combining all kinds of cervical images were benefit for improving the performance of the colposcopy based deep learning model.There are synergistic benefits of the cytology,HPV test,and deep learning model.More accurate cervical screening could be achieved by incorporating the colposcopy based deep learning model,the cytology test results,and the HPV test results.Purpose: Cervical cytology plays an important role in the multimodal cervical cancer screening model.However,in the part II,we used the cervical cytology test report,which was subjectivity,and the results were not reproducible.And cervical cell classification has very important clinical value in the process of cervical cytology screening.This paper aims to classify cervical cells based on a convolutional neural network and propose an improved network that integrates global contextual information and an attention mechanism.Materials and Methods: The experiment is conducted on the SIPa KMe D public data set,which contains a total of 4049 independent cervical cell images in 5 different categories.The backbone network uses an improved Res Net-50 network.The model has one convolutional layer,four convolutional blocks,two pooling layers,four convolutional block attention mechanism modules(CBAM),and a long short-term memory network module(LSTM).Compare the diagnostic performance of our cervical cell multi-classification model with other classical models.Evaluate the loss and accuracy of our model and other models using the cross-entropy loss as the loss function.FLOPs(Floating point operations),Params(total number of parameters to be trained)and average time spent in each epoch were used to evaluate the computational power of the model.Compare the performance of the classifiers with CBAM or LSTM introduced into the backbone network respectively,and the use of CABM and LSTM simultaneously.Results: The experimental results show that the accuracy of the ours model in cervical cell accuracy is 98.89±0.32%,the sensitivity is 99.9±0.10%,the specificity is 99.8±0.10% and the F-measure is 99.89±0.20%,which is better than most cervical cell classification models,which proves the effectiveness of the model.In terms of loss rate and accuracy rate,our classification model converges faster and more stably than other classical models,both on training set and validation set.The FLOPs of our model are smaller than those of the Res Net-101 and VGGNet-16 classification models,respectively.The Params of our classification model are also smaller than those of the VGGNet-16 classification model.And our model introduces the LSTM module,which can reduce useless feature information and reduce model computation time.Conclusion: We proposed a new algorithm of cervical cancer cell classification based on deep learning,which is used to automatically classify cervical cells.Our model is based on the improved Res Net-50 network as the backbone network and using the characteristics of the pyramid pooling layer,the CBAM and the LSTM to extract useful information more effectively.