Agent Step Value (ASV), a replay framework that scores before/after states with a state-less LLM evaluator over a fixed candidate set, turns prompt sensitivity into a measured channel effect and localizes the largest rationale-conditioned losses to extraction and audit.
Agent Step Value (ASV), a state-transition measurement framework that scores each observed action by the change it induces in a state-grounded evaluator’s distribution over fixed candidate outcomes, is introduced.
CARVE is introduced, a black-box counterfactual probe that compares answer changes under matched SHAM and DESTROY replays, and shows only a weak association with annotated temporal coverage, so CARVE is best understood as a routing signal rather than a direct grounding classifier.
Rama Y. AlHamidi, Rasul Khanbayov, E. Serpedin et al.· 0 citations
An auditable framework is built that maintains an external belief state over hidden roles, logs belief updates and belief-action deviations as structured evidence, and supports a defensive offline improvement loop that reviews bad cases before any strategy change.
Yuanpeng Gao, Jiangyi Yang, Yao Zhao et al.· arXiv.org· 0 citations
This work forks live SWE-bench agent trajectories at controlled points, rebuild the environment, continue each fork with a different model, and compare against same-model control forks that isolate sampling and replay noise, finding configuration-dependent temperature-0"determinism" is configuration-dependent.
It is argued that judge evaluations must stratify recall by outcome survival, and release the environment, the injector, all raw verdicts, and an analysis pipeline that rebuilds every number offline.
Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction. Their scores support capability claims only when the evaluation protocol keeps the intended capability necessary for success. Recent reward-hacking benchmarks and system reports show that agents can inst...
