Development And Validation Of Ultrasonography-based Diagnosis And Management System For Thyroid Nodules

Part 1.Validation of the diagnostic performance of the Chinese Thyroid Imaging Reporting and Data Systems(C-TIRADS)Objectives: In 2020,the Society of Ultrasound in Medicine of the Chinese Medical Association developed Chinese Thyroid Imaging Reporting and Data Systems(TIRADS)(C-TIRADS).Here we aimed to validate the performance of C-TIRADS and compared it with the American College of Radiology TIRADS(ACR-TIRADS)and European TIRADS(EU-TIRADS).Methods: This retrospective study included 3438 thyroid nodules(≥10 mm)in 3013 patients(mean age,47.1 years ± 12.9)between January 2013 and November 2019.Ultrasound features of the nodules were evaluated and categorized according to the lexicons of the three TIRADS.We compared these TIRADS by using the area under the receiver operating characteristic curve(AUROC),the area under the precision-recall curve(AUPRC),sensitivity,specificity,net reclassification improvement(NRI),and unnecessary fine-needle aspiration biopsy(FNAB)rate.Results: Of the 3438 thyroid nodules,707(20.6%)were malignant.C-TIRADS showed higher discrimination performance(AUROC,0.857;AUPRC,0.605)than ACR-TIRADS(AUROC,0.844;AUPRC,0.567)and EU-TIRADS(AUROC,0.802;AUPRC,0.455)(P < 0.01).The sensitivity of C-TIRADS(85.3%)was lower than that of ACR-TIRADS(89.1%)but higher than that of EU-TIRADS(78.4%).The specificity of C-TIRADS(76.9%)was similar to that of EU-TIRADS(78.9%)and higher than that of ACR-TIRADS(69.5%).The unnecessary FNAB rate was lowest with C-TIRADS(21.2%),followed by ACR-TIRADS(41.7%)and EU-TIRADS(58.3%).C-TIRADS obtained significant NRI for recommending FNAB over ACR-TIRADS(19.0%,P < 0.001)and EU-TIRADS(25.5%,P < 0.001).Conclusions: C-TIRADS may be a clinically applicable tool to manage thyroid nodules,which warrants thorough tests in other geographic settings.Part 2.Development and validation of an interpretable machine learning diagnosis system based on thyroid ultrasonographyObjectives: Current risk stratification systems for thyroid nodules suffers from low specificity and high biopsy rates.Recently,machine learning(ML)is introduced to assist thyroid nodule diagnosis but lacks interpretability.Here we aimed to develop an interpretable online ML risk stratification system for thyroid nodules and to validate its performance.Methods: This retrospective study included 3965 thyroid nodules(≥10 mm in diameter)diagnosed by FNAB or surgical pathology between January 2013 and December 2018,of which3098 thyroid nodules from center 1 were randomized to the training set(n = 2168)and the internal validation set(n = 930)at a ratio of 7:3.867 thyroid nodules from center 2 of thyroid nodules were used as an independent external validation set.Patient clinical and laboratory test data,including age,gender,underlying thyroid disease and thyroid hormone levels,were collected from electronic medical records.Ultrasound images were evaluated by two sonographers with more than 10 years of experience,based on the ACR-TIRADS guidelines for a total of five ultrasound features: nodule composition,echogenicity,morphology,margins,and echogenic foci,and measurement of the maximum diameter of the nodule.Three machine learning algorithms,Random Forest(RF),Support Vector Machine(SVM)and Extreme Gradient Boosting(XGBoost),were used to construct machine learning diagnostic models based on ultrasound features and ultrasound + clinical features,respectively.The ML models and ACR-TIRADS were compared regarding diagnostic performance and the reliability of guiding FNAB,using the Receiver Operating Characteristic(ROC)curve and the unnecessary biopsy rate.The SHAP(SHapley Additive ex Planation)algorithm is used to resolve the diagnostic process of the optimal model and provide global and individual-level interpretability.Results: Among the included thyroid nodules,2880(72.6%)were benign and 1085(27.4%)were malignant.The ultrasound feature-based RF model(named Thy-Wise)showed the best diagnostic efficacy among all ML models.Compared with ACR-TIRADS,Thy-Wise achieved higher AUC(internal validation set: 0.905 vs.0.857,P < 0.01;external validation set: 0.892 vs.0.853,P < 0.01),accuracy(internal validation set: 82.7% vs.74.8%;external validation set:82.8% vs.73.4%)and specificity(internal validation set: 79.7% vs.68.3%;external validation set: 79.7% vs.66.9%),and significantly reduced unnecessary FNAB rate(internal validation set:14.5% vs.56.6%;external validation set: 14.8% vs.48.3%)while maintaining similar sensitivity(internal validation set: 90.2% vs.91.2%;external validation set: 91.1%).The diagnostic efficacy and the reliability of guiding FNAB of all ultrasound feature-based ML models were not improved by adding clinical features.The SHAP feature importance ranking plot shows that the margin and echogenic foci are the two most important features in Thy-Wise diagnosis,followed by nodule shape,echogenicity,and composition.The SHAP force plot provides a visualization of the direction and degree of influence of each feature of nodules on the diagnostic results of the model.Conclusions: The Thy-Wise model can effectively improve the accuracy of ultrasound diagnosis of thyroid nodules and the reliability of recommended FNAB.Enabling the interpretability of machine learning models helps clinicians and patients to better understand and accept the diagnostic results of machine learning models and helps to facilitate their clinical translation.