Skip to content
Preprint

Repairs for Whom? Evaluator-Executor Mismatch in Selecting Repairs for Failed LLM Agents

Oct 2026 · 0 citations · 41 references
Computer Science

TL;DR

This work labels every candidate with independent rollouts of the deployed policy and measures how much deployment value each evaluator's choice loses, asking whether these evaluators pick repairs that help the agent that will actually execute them.

Abstract

When an LLM agent fails, repair methods choose which action to replace and what to put in its place. They often score candidate repairs with greedy replay, a stronger model, an environment oracle, or a judge that reads the failed log. We ask whether these evaluators pick repairs that help the agent that will actually execute them. Holding failed trajectories and candidate single-step repairs fixed, we label every candidate with independent rollouts of the deployed policy and measure how much deployment value each evaluator's choice loses. On failures not screened for repair value, pre-specified tests show that greedy replay loses 0.08-0.13 in expected net success gain in ALFWorld and 0.06-0.13 in net reward in WebShop, relative to selection with the deployed agent's own rollouts. At an equal rollout budget, selecting repairs for Qwen2.5-7B with Qwen2.5-14B rollouts loses 0.25 in ALFWorld and 0.07 in WebShop, twenty 14B rollouts per candidate are worth no more than two of the 7B agent's own, and fresh rollouts confirm that the two models prefer opposite repairs on selected disagreement cases. The agent's own fine-tuned successor is a worse evaluator still: after one round of training on verified repairs, its rollouts lose 0.55-0.60 when they select repairs for the agent it was trained from, as a pre-specified test confirms. Relative to what selection is worth, substitute models forgo 11-53%; on $\tau^2$-bench, where selecting is worth only 0.10, they lose 0.03. Repairs should be verified with the executor that will run them, and repair labels should name their executor, including its training version.

View source

Similar papers

Book Open access Oct 2026

R²: Repairability-Preserving Recursive Repair for Multimodal Agentic Systems

In an authored study of four incident families, five policies, and 50 seeds, every policy fixed the visible incident and R² audits whether the repaired system can detect, localize, repair, and verify a different unseen fault within budget within budget.

Dominic Dabish · 0 citations
Preprint Oct 2026

Does Failure Feedback Direct Agent Self-Modification?

Self-improving agents use failure feedback to revise their code or instructions, but gains after selection do not establish target-specific repair or generalization. We measure directedness by comparing repair of a randomly selected target failure with repair of the parent's other failures. Across 145 eligible modifica...

Rong He · 0 citations
Preprint Nov 2026

Rethinking Self-Repair in Code-as-Policy Agents: What Drives Retry Success?

Code-as-policy agents often use execution feedback to retry failed programs. Additional sampling and task guidance can also improve success, leaving the contribution of failure-specific feedback unclear. We study this question in simulated robot manipulation. A fixed lookup selects feedback templates using failure labe...

Siddharth Aphale, A. Singh · 0 citations

Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures

It is proved that certifying exact population ties is impossible in finite time, and that the assignment of outcomes to checkpoints carries information beyond marginal outcome distributions, and proposed four diagnostic quantities that expose this failure mode with no additional data collection.

Dong Xu, Zhang-Fan Yang, Jian-Tao Wu et al. · 0 citations
Preprint Nov 2026

How Should Planning Agents ReSHAPE Their Search Space?

Planning agents and benchmarks treat a user's initial specification as fixed. Yet the requests users write exclude plans they would gladly take, since they rarely discover the possible trade-offs they would accept until seeing the options. Agents already revise requests through interaction, but whether this helps plann...

Yi-Lin Zhang, Xin-Ran Zhao, Tong Chen et al. · 0 citations
#artificial intelligence Preprint Oct 2026

When Does Test-Time Search Help World Action Models? An Empirical Audit of Candidate Selection and Simulator Integrity

World-action models increasingly apply test-time scaling, spending extra inference computation on candidate selection and adaptive allocation to improve control. Such gains are credible only when candidates carry exploitable signal, nominal compute savings become real speedup, and evaluation stays reliable. We audit al...

Jun Liu, Da-Jie Sun, Zhenglun Kong et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.