The Design And Implementation Of Judicial Data Quality Oriented Model Evaluation System

With the construction of China's Wisdom Court and the gradual demonstration and implementation of judicial transparency and trial openness,a large number of judicial documents have been exposed to the public and accepted the supervision of the whole society.However,due to the differences in judicial systems around the country,the diversity of case documents,the complexity of trial process and some human errors,the quality of the format and content of the judgment documents varies.A large number of judicial documents not only provide the key information of the case trial,but also serve as the data basis of the study of judicial cognitive intelligence.At present,a large number of models have been accumulated in the field of judicial research,but the proportion of algorithms used in actual implementation is very small,mainly because they are not fully and effectively evaluated.Since a large amount of data is needed,even relatively small errors in training data can lead to large-scale errors in output of the system.The implementation of model evaluation system for data quality can select more appropriate models according to the data quality situation,so as to improve the performance of model training and the accuracy of future judicial decisions.After a comprehensive analysis of the background,current situation and system requirements,this paper uses the Django framework to design and implement a judicial data quality oriented model evaluation system,which is mainly divided into data interaction,document analysis,quality detection and model evaluation modules.Through this system,users can select or upload models and the judicial data sets,carry out judicial intelligence classification tasks such as fine prediction and law articles prediction,and calculate the model evaluation index.In consideration of accuracy,discrimination and stability,the model evaluation indexes used in the system are Accuracy,F1-score,KS value and PSI.For the uploaded judgment documents,the system will automatically perform field parsing,label classification and feature extraction,and generate quality inspection reports.The quality attributes include interpretability,relevancy,accuracy and consistency.Users can choose to configure random scrambling,random deletion,random insertion and other text mutation methods,so that the system can generate a new case description text,and then measures its quality attributes,analyzes the new text and calculates the evaluation index of the judicial classification task model.The system can generate an interpretable visual chart of the correlation between four quality dimensions and four model evaluation indicators under different judicial classification tasks.From the interpretability point of view,the machine learning models KNN,SVM and Naive Bayes are tested as the evaluation objects,and the 20,000 traffic civil first instance judgment documents from China Judgements Online are used as the data set for the experiment of law articles prediction and fine prediction.The experimental results show that in the case of this data set,the Naive Bayes model is more sensitive to the accuracy of data,SVM model is more sensitive to the interpretability,and KNN model is not sensitive to the quality attributes of the judgment documents.Therefore,when the quality of judicial documents can not be guaranteed,KNN model can be used to predict fine and law articles;when the interpretability of judicial documents is higher,SVM model is recommended.Although Naive Bayes is sensitive to accuracy,its classification results are underwhelming.The judicial data quality oriented model evaluation system proposed in this paper has the functions of upload,management and configuration,which is convenient for users to use.It can effectively evaluate the quality of judicial data sets,more comprehensively evaluate the model for performing judicial classification tasks,and gives reasonable suggestions for the model selection under the conditions of different orders of magnitude and quality of judicial documents.It is helpful to improve the accuracy of judicial intelligence task.