Sep 2026· International journal of software engineering and knowledge engineering· 0 citations
Software Testing and Debugging Techniques
TL;DR
An LLM-based, mutation testing-driven approach for test case generation by integrating the semantic understanding of large language models with the precise evaluation mechanism of mutation testing, paving a new path for intelligent test case enhancement.
Abstract
High-quality test cases are crucial for ensuring software reliability. However, automatically generating test cases that can precisely detect semantic defects remains challenging. Mutation testing, by evaluating a test suite's ability to detect artificially seeded faults (mutants), provides an effective benchmark for measuring and improving test quality. Nevertheless, leveraging the results of mutation testing, especially surviving mutants, to automatically guide test case generation and enhancement faces two major obstacles: 1) the semantic understanding gap-inferring high-level fault semantics from low-level syntactic variations of surviving mutants; and 2) the context integration challenge-generated test cases must align with the project's existing coding style, testing framework, and domain-specific knowledge. To address these challenges, this paper proposes an LLM-based, mutation testing-driven approach for test case generation. Our method first employs an Analysis Agent to perform semantic clustering and fault attribution on surviving mutants, identifying genuine fault patterns. Subsequently, a Generate Agent produces targeted test cases that are stylistically consistent and domain-relevant, leveraging the context of the original code and existing test suite. Finally, a Check Agent validates the effectiveness of the newly generated cases. We conducted experiments on a specialized code dataset from the coal industry. The results demonstrate that our approach achieves an accuracy of 84.41% in identifying the causes of surviving mutants, a passing rate of 81.71% in generated test cases, and eliminates 55.66% of the surviving mutants, significantly enhancing the fault detection capability of the test suite. This work paves a new path for intelligent test case enhancement by integrating the semantic understanding of large language models with the precise evaluation mechanism of mutation testing.
This work proposes test-aware mutant generation, in which an LLM receives the problem statement, canonical solution and base tests in a single prompt, and must generate a nontrivial mutant that passes the base unit tests.
Nils Kiele, Zainab Saad, Zi-Rui Wang et al.· International Symposium on E...· 0 citations
This work introduces Mubric, a mutation testing-guided approach to rubric generation, which achieves the highest overall evaluation accuracy, outperforming the strongest baseline by 7.48 percentage points.
Jiayuxuan Yang, Jie M. Zhang, Yiling Lou et al.· 0 citations
Reliable assessment of LLM-generated tests should treat executability as a gate and combine coverage with mutation testing and structural quality indicators, and in practice, model selection should precede prompt tuning.
Bilal I. Al-Ahmad, M. Harshvardhan, Khaled El-Fakih et al.· arXiv.org· 0 citations
An empirical study involving 5 Large Language Models and 4 benchmarks evaluates the effectiveness and efficiency of 3 widely used adequacy criteria: statement coverage, branch coverage, and mutation testing, finding that mutation testing only marginally outperforms traditional coverage criteria in both triggering and d...
Asma Hamidi, Michael Konstantinou, R. Degiovanni et al.· 0 citations
The primary goal of mutation testing is to assess the quality of an application’s test suite. This is accomplished by introducing syntactic changes into a program and determining if any test failures occur for the resulting mutated program, commonly referred to as a mutant. If so, the mutant is said to be killed, confi...
Fari Khalili, Aidan Domondon, Harshit Garg et al.· Proceedings of the ACM on So...· 2 citations
Large language models (LLMs) have demonstrated significant potential for airborne embedded code generation, yet existing evaluation methods based primarily on the Pass@k metric focus on functional correctness and fail to adequately assess robustness degradation under semantic perturbations or identify domain-specific c...
What's new in research at Microsoft? Host Anson Ho explores the latest, including AI-powered universal translation, a breakthrough in a 31-year-old mathematics challenge, and advances making AI coding agents faster and more efficient. The post Microsoft Research Focus – Edition #98 appeared first on Microsoft Research.
AI agents can generate code at a scale that’s hard for humans to review line-by-line. PDD Kit helps ensure the code is doing what the developer intended by creating formal specifications and testing against them. It converts developer intent into a Verus specification, proves AI-generated code meets it for every input, and uses tests where proofs aren't practical. Developers review intent, not every line. The post Formally verifying agent-written full-stack web apps with PDD Kit appeared first o…