ROC Analysis Method Based On K-nearest Neighbor Classifier

The receiver operating characteristic(ROC)curve and the area under the ROC curve(AUC)are indispensable figure of merits for evaluating a binary decision model.ROC analysis originated from the radar signal detection problem in World War II,was used to describe the relationship between true positive rate and false positive rate.Similarly,ROC surface and the volume under the surface(VUS)were introduced for trichotomous tasks.With the characteristics of being insensitive to sample distribution and classification error cost,ROC analysis has been widely used in medical decisionmaking,biomedical information,signal processing,machine learning and others.Although ROC analysis has outstanding performance on classification tasks,there are still a series of problems in its applications.First of all,for a discrete classifier whose output is a class label instead of a continuous value,there is only a single point in ROC space,which makes it impossible to apply ROC curve in evaluating the performance of a discrete classifier.In this paper,basic ideas and methods for ROC analysis of discrete classifiers are introduced.Specifically,K-nearest neighbor classifier was taken as an example for describing ROC analysis of discrete classifiers.Secondly,AUC,VUS and their variances are usually used in practical application as a numerical merit to evaluate the performance of a classifier.Traditional methods of calculating AUC,VUS and their variance have high computational complexity.For example,the algorithm has a square-order computational complexity for AUC calculation and a cubic-order computational complexity for its variance.This makes ROC analysis ineffective to apply in classification tasks with large numbers of samples.In this paper,a bootstrap method which is based on geometry is proposed for calculating AUC and VUS of K-nearest neighbor classifier.The computational complexity for calculating AUC and VUS is linear-order and square-order respectively.More importantly,the computational complexity of the proposed method is independent of the sample size,but is related to the parameter k of the K nearest neighbor.The simulation results show that the proposed method has almost the same performance with the unbiased algorithm,but with less execution time.Most of the applications of ROC are focused on evaluating classifier performance.In order to explore more application scenarios of ROC analysis,this paper try to use ROC analysis to find out an optimal parameter of K-nearest neighbor classifier.Besides,AUC is used as a tool in detecting a change point for fault diagnosis.The experimental results show that the ROC analysis has outstanding performance in the above two application scenarios.