Influence Analysis Of Data Resampling On The Evaluation Of Classification Performance Of ResNet Model

In recent years,the Residual Neural Network Model(ResNet)has been widely used in the fields of image,speech,and text in artificial intelligence,and its classification performance is usually evaluated by data resampling methods,such as Hold-out,RLT,Kfold cross-validation,m×2 RCV(random m×2 cross validation),m×2 BCV(blockregularization m×2 cross validation),and etc.However,the resampling methods are more sensitive to the data partitioning method,and different partitioning ratios and strategies will lead to different performance estimation results,making less reproducible conclusions.Therefore,this thesis investigates the impact of different data resampling methods on classification performance evaluation for ResNet models on an image data set based on the commonly-used ImageNet and CIFAR10 repositories.In this thesis,we propose to use a reproducibility as the evaluation metric for model performance evaluation and formalize the definition of reproducibility and obtain a theoretical lower bound of the reproducibility which shows that the key point to improve the reproducibility is increasing the signal-to-noise ratio of a data-resampling estimator of the model performance.We further investigate variances and the signal-to-noise ratios of the majority-voting estimator and the macro-average estimatitor of the accuracy of a ResNet model.Sufficient experiments are conducted,and the experimental results provides the following conclusions.(1)For a Hold-Out validation with a fixed test set,it is difficult to obtain a reasonable variance estimator of the accuracy estimation.Moreover,With an incrasing size of a training set,the value of the accuracy estimator of a ResNet model increases and tends to 100%gradually.The observation reveals an important weakness of the typical data partitioning in the evaluation of a deep learning classification model.(2)For RLT and K-fold cross validation,the expectation and variance of the accuracy estimator of a ResNet model increase with an increasing data partitioning ratio,and nevertheless the corresponding signal-to-noise ratio decreases.Thus,when partitioning ratio is 5:5,the signal-to-noise ration of an accuracy estimator reaches its maximum value.(3)Under m×2 RCV and m×2 BCV,the trends of the signal-to-noise ratio of accuracy and reproducibility are approximately the same.A higher value of the signal-to-noise ratio indicates a better reproducibility.Moreover,attributing to the constraints of the partitioning of an m×2 BCV,the reproducibility of an m×2 BCV is better than that of m×2 RCV.As long as the number of repetitions m increases,the reproducibility increases.Moreover,the majority-voting estimation method achieves a higher signal-to-noise ratio and a better reproducibility than the macro-average estimation method.In sum,we formalize the definition of reproducibility and provide its theoretical lower bound.We further demonstrate that the lower bound of reproducibility increases as the signal-to-noise ratio of a model performance estimator increases.Furthermore,for the image classification task based on a ResNet model,we recommends the use of a majority-voting based m×2 BCV method to evaluate a ResNet classifiation model.