Development And Validation Of A Model To Predict Tumor Anti-PD-1/PD-L1 Therapy Related Thyroid Dysfunction

【Objective】Anti-PD-1/PD-L1 therapy for malignant tumors has brought clinical benefit,but it is accompanied by specific adverse effects,including thyroid dysfunction,and longterm irreversible damage often cause treatment discontinuation in some patients.How to identify high-risk individuals early and treat in time becomes a big clinical challenge.This study aimed to develop a machine learning prediction model to predict thyroid dysfunction after anti-PD-1/PD-L1 treatment in cancer patients,then validated externally,and the best will be visualized as a nomogram or presented with variable importance.【Methods】This is a retrospective cohort study.Patients with malignant tumors who underwent anti-PD-1/PD-L1 therapy in Yunnan Cancer Hospital from January 2015 to September 2022 were collected.1 Cohort:1.1 Training queues: Cases from January 2015 to March 2022.1.2 Validating queue: Cases from April 2022 to September 2022.2 Group:2.1 Thyroid dysfunction group: TSH occurred during treatment below or above the normal range.2.2 Non-thyroid dysfunction group: TSH is normal.3 Variable screeningUnivariate Logistic analysis was performed at first,then the variables which Pvalue is ≤0.05 entered multivariate logistic regression for variable screening.4 Model development and validationWe trained the models by machine learning methods which including LR,SVM,RF,XGBoost,respectively.The models were internally validated with a five fold crossover in training queues,and were external verified in validating queues.The performance of the models were evaluated by the AUC value,F-1 score,accuracy,sensitivity,and specificity.Calibration was valued by Brier score,calibration curve.5 Model visualizationThe best model was visualized,LR model would be visualized by nomogram;XGBoost and RF model would be showed by the important variables,and the variables were ranked by caculating of the SHAP absolute value.【Results】1.Cohort: There were 464 cases in development queue,and 154 cases in validation queue.2.Group: Thyroid dysfunction group had 83 cases,and 535 cases in non-thyroid dysfunction group.3.Variable screening: Multivariate Logistic regression analysis variables screened out 8 variables: history of alcohol consumption,age,platelets,alkaline phosphatase,thyroid nodules,number of treatment lines,Tg Ab,hormone.These variables were trained by machine learning.4.Machine learning model evaluation and verification4.1 LR modelInternal validation: The AUC is 0.806(95% CI: 0.676-0.936);if the positive prediction rate threshold is set to 0.180,the sensitivity will be 0.802,the specificity be0.751,the accuracy be 0.790,and the F-1 score be 0.503,respectively.Brier score is0.073.External validation: The AUC is 0.828(95% CI: 0.691-0.965);if the positive prediction rate threshold is set to 0.23,the sensitivity will be 0.824,the specificity be0.796,the accuracy be 0.786,and the F-1 score be 0.559,respectively.Brier score is0.073.4.2 SVM modelInternal verification: The AUC is 0.773(95%CI: 0.634-0.914);If the positive prediction rate threshold is set to 0.15,the sensitivity will be 0.756,the specificity be0.754,the accuracy be 0.780,and the F-1 score be 0.474,respectively.Brier score is0.091.External verification: The AUC is 0.827(95% CI: 0.689-0.965);If the positive prediction rate threshold is set to 0.20,the sensitivity will be 0.824,the specificity be0.810,the accuracy be 0.838,and the F-1 score be 0.39,respectively.Brier score is0.089.4.3 RF model Internal verification: The AUC is 0.824(95%CI: 0.706-0.942);If the positive prediction rate threshold is set to 0.151,the sensitivity will be 0.770,the specificity be0.794,the accuracy be 0.806,and the F-1 score be 0.517,respectively.Brier score is0.083.External verification: The AUC is 0.818(95% CI: 0.686-0.950);If the positive prediction rate threshold set to 0.169,the sensitivity will be 0.824,the specificity be0.824,the accuracy be 0.786,and the F-1 score be 0.450,respectively.Brier score is0.083.4.4 XGBoost modelInternal validation: The AUC is 0.840(95%CI: 0.724-0.957);If the positive prediction rate threshold is set to 0.319,the sensitivity will be 0.802,the specificity be0.819,the accuracy be 0.793,and the F-1 score be 0.526,respectively.Brier score is0.081.External verification: The AUC is 0.769(95% CI: 0.630-0.908);If the positive prediction rate threshold is set to 0.310,the sensitivity will be 0.588,the specificity be0.942,the accuracy be 0.708,and the F-1 score be 0.308,respectively.Brier score is0.108.【Conclusion】1.This study has found 8 independent influencing factors of thyroid dysfunction after anti-PD-1/PD-L1 treatment.2.Four models of thyroid dysfunction risk after anti-PD-1/PD-L1 treatment has been successfully constructed.The risk prediction ability of LR,SVM,RF and XGBoost models in the development queue and validation queue is comprehensively evaluated,and the results show that the LR model has good prediction performance in both internal and external validation,and LR has the best risk prediction ability in external verification,which is more suitable for clinical use.