Automatic Software Testing Via Mining Software Data

Software testing has been the primary way for assuring high quality of software systems. In software testing, testers need to design a set of test cases, including test inputs and expected outputs, to cover most of the code and find most bugs before releasing the software. It is challenging to generate such effective test cases manually for complex software systems nowadays. Automatic software testing reduces the laborious human effort in testing. However, the existing test input generation and test oracle generation techniques often require manually provided specifications, which may not be available.;In this thesis, we propose to mine specifications from various kinds of software data, such as the source code, the execution trace, and the existing outputs, to generate test inputs and to verify test outputs automatically.;First, we mine relevant APIs from source code to guide random unit-test generation. Given a method under test (MUT), a key component of object-oriented unit-test generation is to find method-call sequences that create and mutate desired inputs, including the receiver and the arguments. We present a MUT-aware sequence recommendation approach to improve the effectiveness of random object-oriented unit-test generation. Our approach mines relevant APIs for the MUT, i.e. methods that may mutate object fields accessed by the MUT, from the source code. It then recommends short sequences that contain some relevant APIs as the inputs of the MUT. In this way, our approach generates test inputs that have a high chance of exploring most behaviors of the MUT.;Second, we mine potential rules of control branches and variables from execution traces of unverified tests to select tests for result inspection. Our approach collects branch coverage and data value bounds at runtime, and then mines implication relationships between branches and constraints of data values as potential test oracles after running all the tests. Our approach then selects only tests that violate the mined test oracles for result inspection. Experimental results show that our approach can more effectively reduce the number of tests for result inspection while revealing most of the faults.;Third, we mine test oracles of Web search engines from existing inputs (Web queries) and outputs (search results). It is labor-intensive to judge the relevance of search results for a large number of queries, and these relevance judgments may not be reusable since the Web data change all the time. To address this problem, we propose to mine implicit relationships between queries and search results, e.g., some queries may have fixed top 1 result while some may not, and some Web domains may appear together in top 10 results. We define a set of properties of queries and search results, and mine frequent association rules between these properties as potential test oracles. Experiments on major search engines show that our approach mines many high confidence rules that help to understand search engines and detect suspicious search results.