Study Of ATR Algorithm Performance Evaluation Method Based On ROC Curve

With the advance and development of Automatic Target Recognition, the ATR system and the ATR algorithm have become more and more complicated due to the complexity of the application environment and the diversity of the application purpose., compared of which the evaluation of ATR technology is relatively less developed and even becomes the bottleneck of the ATR in a certain period. To evaluate the ATR system properly and efficiently has been a common problem concerned by the ATR developer of all nations in the world. Being an important constitution part of the ATR system, the ATR algorithm plays a significant role and the performance of the algorithm even determines the performance of the whole system in a certain degree, all of which makes the evaluation of the ATR algorithm a hot topic.The thesis is organized as follows:The background and significance of ATR performance evaluation and the history and study status in quo of domain and abroad is introduced, the basic definitions of performance evaluation are specified and that the outline of this thesis is concluded in chapter one.In Chapter two, the traditional evaluation measures such as confusion matrix, probability measure, confidence interval and cost function are introduced. The traditional evaluation method based on those measures also carried out in detail. Following that the default and defects making evaluation conclusion less extensive are analyzed.In Chapter three, we firstly talk about the origin, the mathematic explanation of receiver operating characteristic curve (ROC curve) and its application in pattern recognition, data mining, machine learning and medicine. Then the cluster of distance ROC evaluation measure is defined after the introduction of area under the curve (AUC), the scored area under the curve (SAUC) and the average metric distance (AMD) are introduced. All of these measures are defined and the calculation equations are given. In the end the relationship of accuracy and AUC is deduced.The ROC evaluation method based on bi-normal is introduced and the parameters in the model are estimated in chapter four. Through simulation, we calculate the values of the distance performance measures. The four methods of calculating AUC are analyzed and compared with each other. Finally goes on the evaluation of single algorithm and multi-algorithm based on the AUC measure. We summarize the main work and point out prospect of following study of ROC evaluation method in Chapter 5.