Reporting Completeness Survey And Automated Assessment Study On Abstracts Of Meta-analyses In The Field Of Tumor Drug Therapy

Objective: In this study,the reporting completeness of abstracts of meta-analyses in the field of tumor drug therapy was assessed from the perspective of information integrity to investigate the current status of reporting quality of the abstracts and explore potential factors that may affect the reporting quality of the abstracts.Furthermore,the quality assessment models based on deep learning algorithms were built in an attempt to automatically assess the reporting quality of abstracts of meta-analyses.Methods: Journal articles for meta-analyses in the field of tumor drug therapy were retrieved from Pub Med database,and included articles were selected by strict inclusion and exclusion criteria.According to the PRISMA-A statement,the evaluation criteria of reporting quality of abstracts of meta-analyses was constructed.Then the reporting quality of abstracts of included meta-analyses were manually assessed and results of assessment were presented as total scores of reporting quality as well as the reporting rate of items.Meta-analyses were divided into a high reporting quality group and a low reporting quality group using the 75 th percentile of total scores as the threshold.Then the key factors that may affect the reporting quality of abstracts were analyzed through univariate analysis and binary logistic regression analysis.The data on reporting quality assessment of abstracts of meta-analyses published after 2009 were used as the original dataset for modeling.In addition,all abstracts of meta-analyses in Pub Med database were downloaded to train Word2 Vec word vector.The convolutional neural network models were built to realize automated assessment of the reporting quality of abstracts.Machine learning models based on support vector machines(SVM)were constructed to compare with CNN models.Results: A total of 4915 meta-analysis abstracts were included in this study.The average score of each abstract was 7.75±1.95 and the average reporting rate was 55.15%.The results showed that the overall reporting quality of abstracts of meta-analyses in the field of tumor drug therapy was low.The underreported items mainly focused on item 13(Funding,1.04%),item 14(Registration,4.68%),item 6(Risk of Bias,9.40%)and item11(Strengths and Limitations of Evidence,23.80%).The results of univariate analysis showed that among the 12 characteristic factors,a total of 10 characteristic factors had statistically significant differences between the high and low quality groups(P<0.05).Binary logistic regression analysis showed that meta-analyses published after 2013(OR=2.03),with more words in abstracts(OR=2.12),using structured abstracts(OR=1.51),with number of authors ≤7(OR=1.25)and published in non-oncology journals(OR=1.21)had a higher probability of high reporting quality of abstracts(P<0.05).The research results on the automatic assessment of the reporting quality of abstracts showed that the convolution neural network models based on Word2 Vec pre-training word vector proposed in this study had good prediction performances for the reporting quality of items.Among the models built for 13 items,the accuracy,macro-average F1 or weighted-average F1 values of models built for 11 items on the test set reached more than 80%.The accuracy and weighted-average F1 values of models built for items such as Title,Risk of Bias,Synthesis of Results,Description of the Effect,Funding,and Registration reached over 90%.The accuracy and macro-average F1 values of models built for items such as Key Databases Searched and Search Date reached over90%.The models built for Eligibility criteria and Characteristics of Included participants had poor prediction effect,but macro-average F1 values of the two models on the test set also reached more than 70%.Compared with SVM models,the performances of CNN models for each item was better than those of SVM models.Conclusions: This study assessed the reporting quality of abstracts of meta-analysis papers in the field of tumor drug therapy and pointed out the tendency and deficiency of the content of the abstract reporting in this field.The factors that may affect the reporting quality of abstracts were analyzed to provide reference for improving the reporting quality of abstracts of meta-analyses.In addition,the automated quality evaluation models based on convolutional neural network proposed in this study have good performances,which can automate the process of reporting quality assessment of abstracts of meta-analyses.The models are of great significance for assisting manual assessment of reporting completeness of abstracts of meta-analyses.