Assessing, Comparing, and Combining State machine-Based Testing and Structural Testing: A Series of Experiments

A large number of research works have addressed the importance of models in software engineering, mainly in the design and testing of robust software systems, for example in automating the test of software systems. Although models have been proven to be helpful in a number of software engineering activities, such as providing a better medium for communication among designers and customers, there is still significant resistance to model-driven development in many software organizations. The main reason is that it is perceived to be expensive and not necessarily cost-effective. This study investigates one specific aspect of this larger problem. It addresses the impact of using state machines for testing class clusters that exhibit a state-dependent behavior on testing cost effectiveness when compared with structural testing, which is perhaps the most common systematic testing practice. More precisely, it reports on a series of controlled experiments that investigate the impact of state machine-based testing on fault-detection and cost-effectiveness. Based on previous work showing this is a good compromise in terms of cost and effectiveness, this study focuses on a specific state-based technique that is an adaptation of the W-Method for UML state machines: the round-trip paths coverage criterion. Code-based structural testing is compared to round-trip paths testing and their combination is investigated to determine whether they are complementary. A series of four controlled experiments are presented and discussed in this thesis. Differences between the results of the different experiments are highlighted and plausible explanations for these differences are provided. Results show that, even when a state machine models accurately the behavior of the cluster under test, no significant difference between the fault detection effectiveness of the two test strategies is observed. And in all cases, even when the state machine's representation of the cluster's behavior is limited, the two test strategies are significantly more effective when combined. This implies that a cost-effective strategy could be to specify state machine-based test cases early on, execute them when the source code becomes available, and then complete them with structural test cases based on coverage analysis. A qualitative analysis is also presented, where the reasons for undetected faults are investigated and ways to improve the state machine-based testing of source code are derived. As an outcome of this analysis and based on the presented results, this thesis recommends a number of updates to the round-trip paths testing strategy in order to improve its fault-detection effectiveness and lower its cost.