Aug 2026· International Conference on Automated Software Engineering· Vol 33· 0 citations· 55 references
TL;DR
Results support a focused conclusion: LLM-generated review is most useful as complementary semantic guidance when paired with deployment-oriented test selection, rather than as a standalone testing artifact.
Large language models (LLMs) have opened new opportunities for unit test generation, but executable tests do not necessarily reveal real defects. This paper studies how historical real-bug mechanisms can be transformed into executable feedback targets for LLM-based unit test generation. The proposed framework constructs structural and semantic representations of real-bug records, retrieves mechanisms applicable to a focal method, and instantiates them as synthetic bugs that guide iterative test enhancement. We evaluate the approach on method-level real-bug detection tasks from Defects4J and show that mechanism-guided synthetic-bug feedback improves real-bug detection over execution-, coverage-, mutation-, knowledge-, and search-based baselines. The results suggest that organizing real-bug mechanisms as retrievable and executable feedback targets is an effective way to guide generated tests toward bug-triggering inputs and behavioral oracles.
Large Language Model (LLM) pipelines can generate semantically meaningful unit tests with high coverage, but generated tests often fail to compile or execute. Many frameworks therefore use LLM-driven repair loops that iteratively re-prompt on failures until validation succeeds, but these repairs can introduce issues such as hallucinating an invented class solely to make the test pass. In this paper, we study repair-induced issues and effects of prompt strategy across three Java projects. We build a dataset of LLM-generated broken-to-fixed test pairs, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, derive a taxonomy from 117 manually analyzed pairs (three anti-pattern categories, seven scenarios), and compare the performance of prompting strategies through repair rate and post-repair coverage.
Recent work argues that multi-agent LLM frameworks generate better unit tests than a single well-crafted prompt by iteratively refining coverage and cross-checking oracles. We test this claim directly on a 25-class Java system, comparing CANDOR, a multi-agent framework with JaCoCodriven coverage feedback and panel-based oracle review, against one-shot prompt test generation that submits to an LLM every source file and requirement of our example system in one request. We ran both strategies on GPT-4o and Gemini 2.5 Flash, and every resulting test suite is scored against a human-written baseline using PiTest (with 206 mutants). One-shot prompt outperforms CANDOR across the 25 classes combined: one-shot GPT-4o reaches a 69.90% mutation score versus CANDOR's 45.63%, and one-shot Gemini 2.5 Flash reaches 88.35% versus CANDOR's 35.92%. The one-shot prompt strategy ran on Gemini also surpasses the human baseline (83.98%) in about 45 seconds, while CANDOR with Gemini took roughly 50 hours. CANDOR's central problem is its Initialize stage: it considers the source of each class by itself. Consequently, it cannot reliably construct mocked dependency objects and can fail to produce tests that compile for classes with inter-class dependencies. Restricted to the classes it can handle, CANDOR's per-class mutation scores frequently match or beat the human baseline. Hence, CANDOR's weakness is architectural, not generative.
These findings suggest that while AI agents excel at rigorous boundary testing, they lack the “environmental awareness” needed to write stable, hermetic tests.
Preet Jhanglani, Zeel Desai, Vidhi Kansara et al.· Annual International Compute...· 0 citations
Software testing critically depends on test oracles, yet existing test oracles are often incomplete and insufficient for detecting bugs where implementations deviate from their specifications. Meanwhile, despite advances in test oracle construction, existing techniques typically rely on coarse-grained failure signals or require substantial manual effort, and thus remain inadequate for detecting specification-violation bugs. To address these limitations, we propose JavaOracle, a specification-driven approach that leverages large language models (LLMs) to reason over specifications and systematically enhance test oracles. Specifically, JavaOracle consists of three stages. First, it leverages LLMs to analyze specifications and derive additional test oracles that are not covered by existing tests, while integrating a root-cause-guided repair workflow to ensure that the enhanced test cases are syntactically and semantically valid. Second, it designs a multi-agent debate workflow to distinguish previously unknown specification-violation bugs from assertion failures caused by LLM hallucinations, thereby mitigating the impact of hallucinations. Third, unlike existing approaches that stop at bug detection, JavaOracle further automates test case minimization and bug report generation, producing submission-ready reports without manual effort. We evaluate JavaOracle on 3,961 test cases for Java SE API from the OpenJDK. Experimental results on the latest OpenJDK standard library show that JavaOracle substantially outperforms state-of-the-art baselines, including Fuzz4All, ChatAssert, and Randoop. Cumulatively, JavaOracle discovers 45 previously unknown bugs, which have already been confirmed/fixed by developers, with many persisting since their initial implementation. In an entire pipeline running, JavaOracle reduces execution failures to an average of 6 reports per run, achieving 88.9% precision. In contrast, baseline approaches required exhaustive manual inspection to identify only 0, 3, and 2 real bugs, respectively. Further analysis shows that test cases enhanced by JavaOracle achieve high validity, with an execution pass rate of 79.1%, compared to 35.4%, 92.2% (55.5% test cases unchanged), and 77.6% for the baselines. Ablation studies further demonstrate the effectiveness of JavaOracle components, while the automated pipeline significantly reduces manual analysis effort.
Ruifeng Fu, Yingquan Zhao, Meng Wang et al.· SIGSOFT FSE Companion· 0 citations
Complex Java methods remain challenging for automated unit test generation because achieving high coverage and fault detection often requires satisfying branch-specific testing requirements that are not directly visible from a focal method. Recent LLM-based approaches, such as KTester, PANTA, and MUTGEN, leverage project context, static analysis, coverage feedback, or mutation guidance. However, they do not explicitly represent and track individual testing requirements across iterations. As a result, generation may repeatedly target satisfied requirements while overlooking unresolved branches and weak assertions. Existing approaches also optimize structural coverage and mutation effectiveness separately. We present TATG, a tracking-aware LLM-based unit test generation approach. TATG introduces a unified objective representation that captures testing requirements derived from static analysis and dynamic feedback. The representation enables fine-grained tracking of satisfied and unresolved requirements throughout generation. TATG further employs a two-stage workflow: structural rounds improve coverage, followed by mutation-guided hardening rounds that strengthen assertions and improve fault detection. We evaluate TATG on 141 complex Java methods, including the 110 KTester subjects and 31 additional challenging methods. Compared with KTester and PANTA, TATG improves line coverage, branch coverage, and mutation score by 22.15, 20.14, and 37.66 percentage points on average. On a selected subset of focal methods, TATG also achieves performance comparable to a proprietary industrial test generation tool while achieving higher line coverage and mutation score.
Guancheng Wang, Qinghua Xu, Lionel C. Briand· 0 citations