An Empirical Evaluation of Software Fault-Injection Testing: Effectiveness and Performance
Abstract
Mutation testing is a robust method for assessing the efficacy of software test suites; nevertheless, its actual implementation is frequently constrained by substantial computing expenses, tool heterogeneity, and the creation of redundant or hard-to-find mutants. This paper offers an extensive empirical analysis of mutation testing by assessing eight commonly utilized mutation testing tools on a collection of real-world Java benchmark programs. Many tests are performed with high-coverage test data to ensure that all tools are compared fairly. The results show significant disparities among tools in terms of mutation strength and scalability. Some tools produce many mutants and incur higher execution costs, while others prioritize efficiency by producing fewer, harder-to-detect mutants. A quantitative investigation indicates that average mutation scores vary across lightweight and aggressive tactics, and that tools with lower average mutation scores typically yield a greater proportion of resilient mutants. The findings show that mutation testing effectiveness varies significantly across tools, with clear trade-offs between the number of generated mutants, computational cost, and the ability to produce hard-to-kill faults. These results imply that careful tool selection is essential in practice, as choosing an appropriate mutation testing tool can directly impact testing efficiency, scalability, and the realism of fault detection.