Analysis Of Machine Learning Program Error And Research On Its Detection Technology

Nowadays statistical machine learning is widely adopted in various domains such as data mining,image recognition and automated driving.However,software quality assurance for machine learning is still in its infancy,and many recent efforts have been put into improving the quality of training data and trained models.Unlike them,in this paper we focus on potential bugs in the implementation of ML algorithms,including the learner and classifier.Testing machine learning program is quite hard,because these programs have no test oracle.This problem is seldom considered by ML researchers and practitioners,and they tend to not distinguish ML algorithms and the code imple-menting these algorithms.Bugs in code of machine learning programs can lead to a series of consequences,like program crash,training data losing,performance decrease and time cost increasing.In order to provide a reasonable guidance for developers and users of machine learning applications,we propose a series of solutions for testing code quality,focusing on the coding bugs in machine learning program.Our work can be summarized as follows:· We try to understand the ML program errors.We use mutation testing to gener-ate mutants in order to simulate bugs the programmers made while programming.We use statistical hypothesis testing to compare the difference between the perfor-mance of mutant and the performance of golden version and reference.According to the testing results on different combination of data sets,we classify these mu-tants.Some mutants are fatal errors,such as the one causing compilation failure or crash.Some mutants can severely degrade program performance,while others make the program run out of time.In addition,a large number of mutants have not been executed,indicating that code coverage is still meaningful.Finally,two types of mutants are more important:one is statistically equivalent mutant,the other is statistically stubborn mutant.· We try to classify these two kinds of mutants.Statistically equivalent mutant should be treated as correct program while statistically stubborn mutant should be found out because they belong to well-hidden errors.First we use metamorphic testing and find it can detect some of the statistically stubborn mutants,but there is still ex-isting some false negative results.And then we use the property of ML algorithms to distinguish them,including the performance on learning curves and the robust-ness to adversarial examples.We observed that some mutants can be distinguished obviously,but some are still indistinguishable.It indicates that testing ML program is still difficult,due to the lack of effective oracles.· We design and implement MINT system based on above techniques.We evaluate MINT with experiment.The result shows that MINT is capable of guiding program-mers to test ML programs.It can provide some useful information and is convenient for programmers.