Skip to content

LLM-Mestcase: A Test Case Modification Method Based on LLMs and Mutation Testing

Sep 2026 · International journal of software engineering and knowledge engineering · 0 citations
Software Testing and Debugging Techniques

TL;DR

An LLM-based, mutation testing-driven approach for test case generation by integrating the semantic understanding of large language models with the precise evaluation mechanism of mutation testing, paving a new path for intelligent test case enhancement.

Abstract

High-quality test cases are crucial for ensuring software reliability. However, automatically generating test cases that can precisely detect semantic defects remains challenging. Mutation testing, by evaluating a test suite's ability to detect artificially seeded faults (mutants), provides an effective benchmark for measuring and improving test quality. Nevertheless, leveraging the results of mutation testing, especially surviving mutants, to automatically guide test case generation and enhancement faces two major obstacles: 1) the semantic understanding gap-inferring high-level fault semantics from low-level syntactic variations of surviving mutants; and 2) the context integration challenge-generated test cases must align with the project's existing coding style, testing framework, and domain-specific knowledge. To address these challenges, this paper proposes an LLM-based, mutation testing-driven approach for test case generation. Our method first employs an Analysis Agent to perform semantic clustering and fault attribution on surviving mutants, identifying genuine fault patterns. Subsequently, a Generate Agent produces targeted test cases that are stylistically consistent and domain-relevant, leveraging the context of the original code and existing test suite. Finally, a Check Agent validates the effectiveness of the newly generated cases. We conducted experiments on a specialized code dataset from the coal industry. The results demonstrate that our approach achieves an accuracy of 84.41% in identifying the causes of surviving mutants, a passing rate of 81.71% in generated test cases, and eliminates 55.66% of the surviving mutants, significantly enhancing the fault detection capability of the test suite. This work paves a new path for intelligent test case enhancement by integrating the semantic understanding of large language models with the precise evaluation mechanism of mutation testing.

View source

Similar papers

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

This work proposes test-aware mutant generation, in which an LLM receives the problem statement, canonical solution and base tests in a single prompt, and must generate a nontrivial mutant that passes the base unit tests.

Nils Kiele, Zainab Saad, Zi-Rui Wang et al. · 0 citations
Sep 2026

Evaluating the effectiveness of class-level LLM-generated test suites in Python

Reliable assessment of LLM-generated tests should treat executability as a gate and combine coverage with mutation testing and structural quality indicators, and in practice, model selection should precede prompt tuning.

Bilal I. Al-Ahmad, M. Harshvardhan, Khaled El-Fakih et al. · 0 citations
#software testing Preprint Sep 2026

How effective are traditional test criteria at detecting bugs in large language models generated code?

An empirical study involving 5 Large Language Models and 4 benchmarks evaluates the effectiveness and efficiency of 3 widely used adequacy criteria: statement coverage, branch coverage, and mutation testing, finding that mutation testing only marginally outperforms traditional coverage criteria in both triggering and d...

Asma Hamidi, Michael Konstantinou, R. Degiovanni et al. · 0 citations
Open access Oct 2026

LLMutantKiller: Using Large Language Models to Generate Tests That Kill Mutants

The primary goal of mutation testing is to assess the quality of an application’s test suite. This is accomplished by introducing syntactic changes into a program and determining if any test failures occur for the resulting mutated program, commonly referred to as a mutant. If so, the mutant is said to be killed, confi...

Fari Khalili, Aidan Domondon, Harshit Garg et al. · 2 citations
Conference Aug 2026

A Mutation-Driven Trustworthiness Evaluation Method for LLM-Based Airborne Code Generation

Large language models (LLMs) have demonstrated significant potential for airborne embedded code generation, yet existing evaluation methods based primarily on the Pass@k metric focus on functional correctness and fail to adequately assess robustness degradation under semantic perturbations or identify domain-specific c...

Tong Zhang, Xing-Zhi Wang, Kang-Fa Xu · 0 citations

Related blog posts

Microsoft Research Blog Nov 12, 2026

Microsoft Research Focus – Edition #98

What's new in research at Microsoft? Host Anson Ho explores the latest, including AI-powered universal translation, a breakthrough in a 31-year-old mathematics challenge, and advances making AI coding agents faster and more efficient.  The post Microsoft Research Focus – Edition #98 appeared first on Microsoft Research.

Microsoft Research Blog Nov 5, 2026

Formally verifying agent-written full-stack web apps with PDD Kit

AI agents can generate code at a scale that’s hard for humans to review line-by-line. PDD Kit helps ensure the code is doing what the developer intended by creating formal specifications and testing against them. It converts developer intent into a Verus specification, proves AI-generated code meets it for every input, and uses tests where proofs aren't practical. Developers review intent, not every line. The post Formally verifying agent-written full-stack web apps with PDD Kit appeared first o…

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.