Threshold-aware Defect Prediction On Imbalanced Datasets

Defect prediction strives to establish a defect prediction model to predict the code regions in the software that contain defects.An accurate defect prediction model can help developers concentrate on files/methods that are more likely to contain defects,so as to better allocate software resources to improve the efficiency of software quality assurance and maintenance activities.The inherent difficulty in the current defect prediction research field is that the defect datasets extracted by real-world software are almost imbalanced,that is,the number of modules with defects is generally much smaller than the number of modules without defects.When training defect prediction models on imbalanced datasets,machine learning classifiers often lead to bias because of the imbalances between the numbers of modules with/without defects,which makes it hard to construct accurate defect prediction models.The states of the arts of rebalancing techniques in the field of defect prediction include dataset rebalancing techniques(e.g.under sampling and over sampling)and ensemble learning(e.g.Bagging methods).However,the rebalancing techniques often bring a series of side effects to the defect prediction model,such as incomplete training and overfitting.In the process of defect prediction,except various dataset rebalancing techniques and model-assisted training methods,similar effects can be achieved by adjusting the threshold.However,in the field of defect prediction,the threshold is often ignored,and the default threshold(0.5)is used to classify the prediction result into two classes.According to the literature survey in this article,few researchers in the current defect prediction research have employed threshold adjustment to optimize the impact of prediction model accompanying with imbalanced datasets.This paper attempts to cut in from the threshold that is largely ignored in the defect prediction process,and tries to find a method that is more suitable for doing defect prediction on imbalanced datasets than rebalancing techniques.To this end,this paper proposes 7 methods of threshold prediction in the process of defect prediction to optimize the model prediction results,threshold-aware techniques.Threshold-aware techniques are calculated adaptively according to the defective ratio on the dataset which will be used for classifying the prediction value as new threshold.In this paper,we do within-project defect prediction on datasets which are selected by the standard of less than 20%defective ratio from three data repositories commonly used in the field of defect prediction research,PROMISE,NASA,and AEEEM.To perceive the applicability of threshold-aware techniques from defect prediction on imbalanced datasets,here we mainly conduct works from the following three scenarios.(a)Threshold-aware defect prediction platform-We launch a defect prediction platform which contains rebalancing techniques and threshold-aware techniques.(b)Performance of threshold-aware techniques-We compare threshold-aware techniques with traditional classifiers and rebalancing techniques to analyze the necessity,effectiveness and extensiveness of threshold-aware techniques.(c)Application of threshold-aware techniques-We find out the applicability of threshold-aware techniques in different situations.Experimental results show that:(a)threshold-aware techniques can greatly improve the performance of traditional classifiers for defect prediction on imbalanced datasets in various indicators(Recall,F1,Gmean etc.);(b)the performance of threshold-aware techniques in most of indicators are better than rebalancing techniques,especially in three indicators Recall,Gmean and Gmeasure;(c)the rebalancing techniques combining with part of threshold-aware methods,can perform better than combing with default threshold when doing defect prediction on imbalanced datasets.Based on the findings above,we draw the following conclusions:(a)we strongly recommend to use threshold-aware techniques when doing defect prediction on imbalanced datasets;(b)threshold-aware techniques should be used instead of rebalancing techniques when doing defect prediction on imbalanced datasets;(c)by properly selecting the combination of threshold-aware method and rebalancing techniques,performance can be improved.