TL;DR
Overall, TraceGate shows that rethinking debugging through controlled observability, rather than relying solely on stronger models or larger prompts, can make LLM-assisted repair more effective, efficient and controllable.
Large language models (LLMs) show promise for vulnerability discovery, yet prevailing methods often inspect code in isolation, struggle with long project contexts, and produce coarse function- or file-level predictions that offer limited guidance for targeted patches. This paper presents a controlled empirical study of what factors help LLM agents perform project-level, line-level vulnerability localization from runtime failure evidence. We introduce T2L (Trace-to-Line), a reproducible research framework that narrows repository-scale code into candidate vulnerable lines through AST-based chunking, structured diagnostic information collection, and evidence-guided refinement. To support systematic evaluation, we build T2L-ARVO, an expert-verified 50-case benchmark spanning five crash families in real-world open-source projects, and T2L-Agent, a modular reference agent for studying localization workflows under equal per-case budgets. Across ten models and controlled ablations, we find that structured diagnostic analysis, coarse-to-fine refinement, and multi-hypothesis exploration all improve trace-to-line localization. Across these workflow-level configurations, the largest observed increase in line-level localization reaches 48.9 percentage points over the corresponding baseline, while increasing reasoning budget or adjusting sampling temperature yields limited and inconsistent benefits. Overall, our study provides an empirical reference for designing LLM-agent workflows for complex project-level localization tasks that move from runtime failure evidence to precise vulnerable-line hypotheses.
ContainmentBench, a sandboxed benchmark comprising a 504-scenario specification dataset, a shared rollout-trace schema, and stage-scoped metrics for endpoint violations, logged propagation, and explicitly authorized taint-exposed proposals that commit, is introduced.
Self-supervised automated program repair (APR) leverages project-specific perturbations to generate training data and uses test execution diagnostics to guide patch generation. In practice, however, diagnostics are heterogeneous (e.g., exception messages, stack traces, assertion diffs, and dynamic execution signals) and must fit within a strict context budget. Naive concatenation either truncates critical evidence or amplifies noisy artifacts, especially for deep bugs where the failure symptom is far from the root cause. We present TraceStructRepair, a diagnostic structuring and budgeting approach for execution-aware self-supervised APR. TraceStructRepair (1) extracts a compact set of execution diagnostics from a single failing test, including exception type and message, stack trace frames, assertion diffs, and optionally dynamically loaded classes; (2) normalizes and ranks diagnostic elements to reduce redundancy and framework noise; and (3) assembles a field-aware representation under a fixed token budget with per-field caps and lexicographic priority rules. We implement TraceStructRepair on top of the SelfAPR pipeline and evaluate it on Defects4J using a project-wise heldout protocol. Beyond end-to-end repair outcomes, we analyze robustness under noisy fault localization and component ablations. We release artifacts to facilitate replication and future work on execution-aware, budget-constrained APR.
DCAware is proposed, a computationally efficient, non-iterative framework that prioritizes high signal-to-noise contextual grounding over multi-round repair and improving contextual quality is more effective than adding iterative repair complexity in the studied regression-oracle setting.
Automated testing discovers vulnerabilities faster than developers can investigate and repair them, leaving an interval in which known memory corruptions remain exploitable. End- to-end LLM repair agents can shorten this interval, but they synthesize open-ended code changes and commonly validate them only by replaying a proof of concept (PoC). This weak oracle accepts patches that silence the observed crash by changing unrelated behavior, making unintended deployment risky. We present CodeMechanic, a bug-property-guided system for generating constrained mit- igations for spatial memory corruption. Instead of asking an LLM to generate a permanent repair, CodeMechanic reconstructs the violated memory-safety property from the crash, validates the dereferenced pointer and its buffer range, and inserts a local fail-stop guard before the dangerous access. The guard terminates execution when the boundary check fails. The resulting mitigation deliberately trades availability for security: it can convert potential remote code execution into controlled termination while developers investigate the root cause and prepare a permanent repair. CodeMechanic combines a two-dimensional static and dynamic context extractor with in-prompt debugging knowledge and stepwise val- idation to limit the effect of LLM errors. On 101 real-world ARVO bugs, the first attempt of CodeMechanic produces 47.6% more plausible patches (i.e., patches that pass PoC- replay validation) than the best baseline while using 91% fewer tokens. Manual audit further shows that CodeMechanic produces 3.4x - 4.3x more patches semantically equivalent to developer-written repairs.
When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect. We measure how often that treatment is warranted. BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) captures each validation command at its exact working-tree state, extracts a test-only patch, and replays the command on the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome and the replay results assign every event an evidence role, from gold-aligned bug-discriminating through regression-only to misleading. Across 3,730 events in 643 rollouts on 110 tasks, 46.0% of positive comparable events carry no bug-discriminating information; 23.8% of baseline rollouts, with no feedback injected, close with a patch whose entire positive evidence base is of this kind. A three-arm experiment tests whether returning the B-replay outcome to the agent changes this pattern. Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p = 0.0029) and raises bug-discriminating evidence by 7.4 points (p = 0.011), with no detectable cost to repair success. Both estimates fall below the prespecified 10-percentage-point smallest effect size of interest, so practical magnitude remains uncertain. Roughly a third of the improvement traces to the reminder alone; across two exploratory replications, varying the scaffold and the model, the B-replay content adds a detectable increment only with gpt-5.6-sol under the unconstrained tool-use loop. BSG-VA applies post hoc to any replayable repair trajectory that preserves the required code states and execution environment. Keywords: program repair agents, validation evidence, test adequacy, large language models, software quality, controlled experiment.