Sep 2026· Brazilian Symposium on Systematic and Automated Software Testing· pp. 274-276· 0 citations· 4 references
TL;DR
The experience of applying mutation testing to Bitcoin Core, the reference implementation of the Bitcoin protocol, shows that mutation testing is tractable even on large, complex codebases when the tooling is designed around the realities of the project, and offers lessons for practitioners considering similar approaches on safety-critical systems.
Abstract
Mutation testing is a technique for evaluating test suite effectiveness, yet its adoption in industrial settings remains limited due to scalability challenges. In this paper, we report our experience applying mutation testing to Bitcoin Core, the reference implementation of the Bitcoin protocol - a safety-critical, C++ codebase with hundreds of thousands of lines of code and a heterogeneous test suite. We present bcore-mutation, an open-source tool written in Rust and tailored to Bitcoin Core’s specific constraints, and describe the challenges we faced: test selection for incremental analysis, unproductive mutant filtering, and surfacing results to pull request authors without disrupting the review workflow.We further discuss how mutation testing was applied to evaluate the quality of fuzz harnesses, and report an experimental use of Large Language Models to automatically generate tests that kill surviving mutants. Our experience shows that mutation testing is tractable even on large, complex codebases when the tooling is designed around the realities of the project, and offers lessons for practitioners considering similar approaches on safety-critical systems.
Mutation testing introduces synthetic defects into a program under test to assess test suite adequacy. While recent advances in large language models (LLMs) have led to more realistic and hard-to-kill mutants, the field is currently dominated by single-purpose research prototypes that tightly couple the mutant generati...
Megan Maton, Gregory M. Kapfhammer, Phil McMinn· Proceedings of the 41st IEEE...· 1 citation
PyMut4SE is a novel mutation tool for Python that focuses on a comprehensive set of mutations for any Python project, providing a rich and extensible set of mutation operators, access to mutated source code and its characteristics, detailed execution and behavioral observations, and support for both selective mutation...
Laura Plein, Matthieu Jimenez, Mike Papadakis· Companion Proceedings of the...· 0 citations
Mutation testing is a robust method for assessing the efficacy of software test suites; nevertheless, its actual implementation is frequently constrained by substantial computing expenses, tool heterogeneity, and the creation of redundant or hard-to-find mutants. This paper offers an extensive empirical analysis of mut...
Bahman Arasteh, H. Kusetogullari, Mehrdad Abdollahian et al.· 2026 International Conferenc...· 0 citations
Mutation testing assesses test suite quality by injecting controlled faults into production code. Recent studies show that large language models (LLMs) can generate more diverse mutants, closer to real faults, than traditional syntactic operators, though at the cost of more non-compilable, duplicate, and equivalent mut...
Victor Manuel Chang, Marcelo Manoel de Lima, Awdren de Lima Fontão· Brazilian Symposium on Syste...· 0 citations
Evaluating the generalization capabilities of the proposed LLM-based EMD techniques across two additional datasets that differ from the prior dataset in mutation operators, programming languages, or source projects finds that the studied LLMs tend to rely on a method-wise majority-voting shortcut rather than reasoning...
Arjun Tandon, Mehmet Fırat Dündar, Milkiyas Gebremichael Gebru et al.· Proceedings of the ACM on So...· 0 citations
This work proposes test-aware mutant generation, in which an LLM receives the problem statement, canonical solution and base tests in a single prompt, and must generate a nontrivial mutant that passes the base unit tests.
Nils Kiele, Zainab Saad, Zi-Rui Wang et al.· International Symposium on E...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.