Jul 2026· International Conference on Artificial Intelligence Testing· pp. 60-67· 0 citations· 11 references
Abstract
Research on LLM-based code generation has achieved strong performance on function-level benchmarks, yet repository-level code generation in real-world projects still faces challenges from cross-file dependencies. Existing studies usually emphasize single-round or final-round outcomes, which limits the analysis of where feedback gains concentrate and when later repair begins to offset earlier progress. We propose a dynamic evaluation framework for repository-level code generation based on DevEval’s Python tasks. The framework constructs processed test sets, evaluates multi-round repair trajectories with unit-test feedback over multiple rounds, and characterizes trajectories via round-level effectiveness, feedback-induced gain, and regression stability. However, roughly three-quarters of the gains emerged within the first three rounds, and net feedback gain turned negative by Round 5. Previously repaired errors reappeared in later iterations, with reappeared error rates from 4.5% to 8.8% across models. These findings indicate that unit-test feedback is most effective as an early-stage signal, and that practical repair systems should adopt early-round budgeting with best-so-far retention rather than open-ended loops.
MCR-Bench is introduced, the first defect state-aware benchmark designed for realistic multi-round code review, and in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.
De-Wu Zheng, Yan-Lin Wang, Xi-Wen Wang et al.· 0 citations
Findings indicate that cross-model collaboration offers a practical and parameter-efficient alternative to scaling up monolithic models for code generation and maintains competitive accuracy when only 20% of test cases are available for diagnostic feedback.
Jiangping Huang, Wen-Guang Ye, Weisong Sun et al.· ACM Transactions on Software...· 0 citations
This paper identifies patch verbosity as a major yet overlooked concern in LLM-based APR and proposes RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation that achieves a substantially better size-correctness tradeoff.
Wen-Qiang Luo, J. Keung, Xiaoyu Shi et al.· 0 citations
Test-time refinement aims to improve generated programs through additional inference, but its value after an initial candidate has been produced remains unclear. We conduct a controlled evaluation of one-round Self-Refine and Self-Debug across seven models and three Python code-generation benchmarks. For each model and task, both methods refine the same initial candidate, allowing us to measure refinement gain without variation in initial generation. Self-Debug produces positive refinement gain in 16 of the 21 model–benchmark combinations and no change in the remaining five, whereas Self-Refine reduces correctness in 16 combinations and improves it in only four. Decomposing refinement gain into repairs and regressions clarifies this contrast. After a diagnostic pass, Self-Debug’s candidate preservation leaves 34.1% of initial final-test failures unaddressed, but 99.95% of initially correct candidates remain correct. With regressions nearly absent, repairs after diagnostic failures translate directly into positive gain. Self-Refine also repairs initial failures, but its regression count is more than three times its repair count overall, producing predominantly negative gain. Resource analysis shows that Self-Refine uses more tokens while generally reducing correctness, whereas Self-Debug provides a more favorable gain–overhead balance, although its inference overhead per net additional pass varies across models and benchmarks. These results show that one-round refinement is not inherently beneficial and should be applied only when its expected gain justifies the additional computation and cost.
This paper introduces T2L (Trace-to-Line), a reproducible research framework that narrows repository-scale code into candidate vulnerable lines through AST-based chunking, structured diagnostic information collection, and evidence-guided refinement that improves trace-to-line localization.
An automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok is presented and a substantial gap between correctness and quality attributes is revealed.
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
