Skip to content
Review Open access

Mutation Testing in the Wild: Challenges and Lessons from Bitcoin Core

Sep 2026 · Brazilian Symposium on Systematic and Automated Software Testing · pp. 274-276 · 0 citations · 4 references

TL;DR

The experience of applying mutation testing to Bitcoin Core, the reference implementation of the Bitcoin protocol, shows that mutation testing is tractable even on large, complex codebases when the tooling is designed around the realities of the project, and offers lessons for practitioners considering similar approaches on safety-critical systems.

Abstract

Mutation testing is a technique for evaluating test suite effectiveness, yet its adoption in industrial settings remains limited due to scalability challenges. In this paper, we report our experience applying mutation testing to Bitcoin Core, the reference implementation of the Bitcoin protocol - a safety-critical, C++ codebase with hundreds of thousands of lines of code and a heterogeneous test suite. We present bcore-mutation, an open-source tool written in Rust and tailored to Bitcoin Core’s specific constraints, and describe the challenges we faced: test selection for incremental analysis, unproductive mutant filtering, and surfacing results to pull request authors without disrupting the review workflow.We further discuss how mutation testing was applied to evaluate the quality of fuzz harnesses, and report an experimental use of Large Language Models to automatically generate tests that kill surviving mutants. Our experience shows that mutation testing is tractable even on large, complex codebases when the tooling is designed around the realities of the project, and offers lessons for practitioners considering similar approaches on safety-critical systems.

Read PDF

Similar papers

Book Open access Oct 2026

multiplex: A Modular LLM-Based Mutation Framework

Mutation testing introduces synthetic defects into a program under test to assess test suite adequacy. While recent advances in large language models (LLMs) have led to more realistic and hard-to-kill mutants, the field is currently dominated by single-purpose research prototypes that tightly couple the mutant generati...

Megan Maton, Gregory M. Kapfhammer, Phil McMinn · 1 citation
#software testing Book Open access Oct 2026

PyMut4SE: Comprehensive Mutation Testing for Python

PyMut4SE is a novel mutation tool for Python that focuses on a comprehensive set of mutations for any Python project, providing a rich and extensible set of mutation operators, access to mutated source code and its characteristics, detailed execution and behavioral observations, and support for both selective mutation...

Laura Plein, Matthieu Jimenez, Mike Papadakis · 0 citations
Conference Open access Sep 2026

An Empirical Evaluation of Software Fault-Injection Testing: Effectiveness and Performance

Mutation testing is a robust method for assessing the efficacy of software test suites; nevertheless, its actual implementation is frequently constrained by substantial computing expenses, tool heterogeneity, and the creation of redundant or hard-to-find mutants. This paper offers an extensive empirical analysis of mut...

Bahman Arasteh, H. Kusetogullari, Mehrdad Abdollahian et al. · 0 citations
Open access Sep 2026

LLM-Pyro: LLM-Based Mutation Testing with Equivalence Triage for Python

Mutation testing assesses test suite quality by injecting controlled faults into production code. Recent studies show that large language models (LLMs) can generate more diverse mutants, closer to real faults, than traditional syntactic operators, though at the cost of more non-compilable, duplicate, and equivalent mut...

Victor Manuel Chang, Marcelo Manoel de Lima, Awdren de Lima Fontão · 0 citations
Open access Oct 2026

Re-evaluating Detection of Equivalent Mutants using LLMs: We Should Properly Measure How Far We Are

Evaluating the generalization capabilities of the proposed LLM-based EMD techniques across two additional datasets that differ from the prior dataset in mutation operators, programming languages, or source projects finds that the studied LLMs tend to rely on a method-wise majority-voting shortcut rather than reasoning...

Arjun Tandon, Mehmet Fırat Dündar, Milkiyas Gebremichael Gebru et al. · 0 citations

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

This work proposes test-aware mutant generation, in which an LLM receives the problem statement, canonical solution and base tests in a single prompt, and must generate a nontrivial mutant that passes the base unit tests.

Nils Kiele, Zainab Saad, Zi-Rui Wang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.