Artificial Intelligence Identification Model For Diabetic Retinopathy With Multimodal Images

Purpose:Construct an automatic identification model of diabetic retinopathy(DR)based on deep learning by using multimodal images(confocal laser ultra-widefield fundus photography and optical coherence tomography angiography(OCTA)),and to realize the automatic diagnosis and screening of DR.Methods:1.Application of Transformer to construct DR automatic identification model based on confocal laser fundus photography(1)The subjects of the study were patients diagnosed with DR and met the inclusion criteria in the Department of Ophthalmology of Xiangya Second Hospital of Central South University from October 2021 to February 2022.(2)All patients underwent ophthalmic examination and confocal laser ultra-widefield fundus photography(Confocal Laser Scanning Angiography System,Suzhou Micro Clear Medical Instruments Co.,Ltd,Suzhou).Fundus images of 60° and 100° centered on macular were obtained for each eye.(3)The qualified images were manually segmented and labeled with semantic pixel-level DR lesions by the ophthalmologists using the Colabel software.The annotated lesions included intraretinal hemorrhages,microaneurysms,hard exudates,soft exudates,fibroproliferative membranes,neovascularization,venous beading,venous loop,and intraretinal microvascular abnormality(IRMA).(4)Established the annotation image dataset.(5)Using the labeled dataset to train the modified Transformer semantic segmentation network model to automatically identify DR lesions,and calculated the accuracy(ACC),intersection-over-Union(Io U),sensitivity,specificity,Dice score,and area under the receiver operating characteristic curve(AUC)to evaluate its recognition performance.(6)Compare this model with other network models.2.Application of U-Net convolutional neural network(CNN)model for automatic identification of diabetic macular ischemia(DMI)(1)The study included DR patients with macular ischemia in the Department of Ophthalmology,Second Xiangya Hospital of Central South University from April 2020 to October 2021.During the same period,agematched healthy people were recruited as the control group.(2)All subjects received ophthalmological examination and OCTA detection.(3)The foveal avascular zone(FAZ)boundary on the full-thickness retina of6x6 mm enface OCTA images of DMI and normal eyes were manually labeled by Image J software.(4)Established the annotation image dataset.(5)The U-Net CNN model with two attention modules was trained with the labeled qualified images.The improved U-Net CNN model was used to automatically identify the FAZ in DMI and established an automatic identification CNN model of the FAZ area based on OCTA images.The Dice score,Jaccard index,and area Pearson correlation coefficient were calculated to evaluate the accuracy of the model and compared with the baseline U-Net model.(6)The model was applied to the public dataset SFAZ to compare its effectiveness with existing models in identifying and quantifying FAZ.Results:1.In this study,a new automatic identification model of DR lesions based on Transformer was developed and verified by establishing the data set of confocal laser ultra-widefield fundus photography of DR patients.A total of 1070 confocal laser ultrawide-field fundus photographs were included.ACC and Io U values were as follows,respectively: transformer network model for background recognition,0.991 and 0.987;hemorrhage,0.819 and 0.538;microaneurysms,0.591 and 0.288;hard exudates,0.791 and 0.532;soft exudates,0.782 and 0.563;neovascularization,0.792 and0.606;retinal vascular abnormalities(IRMAs),0.509 and 0.217.ACC for the overall assessment of the image was 0.987,and the mean Io U was 0.533.The model showed superior performance to other network models in identifying DR lesions when tested.In the dataset,sensitivity,specificity,Dice score,and AUC values of 0.987,0.998,0.658 and 0.977 were achieved,respectively.2.In this study,a new CNN model was established to automatically measure the FAZ area of diabetic macular ischemia.The study included110 OCTA images.The Dice score of the FAZ area predicted by the proposed model was 0.948 ± 0.046,the Jaccard index was 0.912 ± 0.069,and the area Pearson correlation coefficient was 0.996 ± 0.004;in the description dataset SFAZ,the Dice score of the proposed model to predict the FAZ area was 0.983 ± 0.005,the Jaccard index was 0.968 ± 0.010,and the area Pearson correlation coefficient was 0.950 ± 0.010,which was overall better than previous studies on this dataset.Conclusions:1.This study developed and verified a new deep learning system based on the Transformer for automatic identification of DR lesions in confocal laser ultra-widefield fundus photography.Compared with traditional fundus photography,confocal laser fundus photography has great advantages in identifying DR lesions,especially IRMA,venous bead,and neovascularization.In this study,the semantic segmentation method based on Transformer was used for the first time to construct the automatic identification model for DR lesions in the ultra-widefield fundus photography,and thus have better performance.2.This study established a new convolution neural network model,which was more accurate than the traditional CNN in automatically measuring the FAZ area on OCTA images through the improved U-Net CNN model.The model can more accurately and automatically measure the DMI index,thereby assisting the diagnosis and prognosis analysis of macular ischemia in vascular diseases.3.The artificial intelligence automatic identification model based on multimodal images is expected to become an effective auxiliary tool for DR diagnosis and treatment.