Understanding the Effect of Agentic Iteration on LLM-Based Unit Test Generation: A Comparison with Human-Written Tests Using Mutation Testing

Abstract

Recent work argues that multi-agent LLM frameworks generate better unit tests than a single well-crafted prompt by iteratively refining coverage and cross-checking oracles. We test this claim directly on a 25-class Java system, comparing CANDOR, a multi-agent framework with JaCoCodriven coverage feedback and panel-based oracle review, against one-shot prompt test generation that submits to an LLM every source file and requirement of our example system in one request. We ran both strategies on GPT-4o and Gemini 2.5 Flash, and every resulting test suite is scored against a human-written baseline using PiTest (with 206 mutants). One-shot prompt outperforms CANDOR across the 25 classes combined: one-shot GPT-4o reaches a 69.90% mutation score versus CANDOR's 45.63%, and one-shot Gemini 2.5 Flash reaches 88.35% versus CANDOR's 35.92%. The one-shot prompt strategy ran on Gemini also surpasses the human baseline (83.98%) in about 45 seconds, while CANDOR with Gemini took roughly 50 hours. CANDOR's central problem is its Initialize stage: it considers the source of each class by itself. Consequently, it cannot reliably construct mocked dependency objects and can fail to produce tests that compile for classes with inter-class dependencies. Restricted to the classes it can handle, CANDOR's per-class mutation scores frequently match or beat the human baseline. Hence, CANDOR's weakness is architectural, not generative.