Skip to content

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

Oct 2026 · 1 citation · ⚡ 1 influential · 50 references
Computer Science

TL;DR

Agent evaluations should be evaluated as configurable systems themselves, and that some desired behaviors are more effectively implemented in the system than requested through prompting, according to a new benchmark of four scientific tasks.

Abstract

Agent evaluations increasingly go beyond a single success rate, reporting metrics such as cost, consistency, and robustness. Yet they typically treat the agent itself as fixed. In practice, an agent is a configurable system: users decide what to tell it, how long to let it run, and which model to use, and each of these choices can change how well and how consistently it performs. We study these choices on a new benchmark of four scientific tasks, where a coding agent must find and correctly operate a published specialist model. We investigate five parts of the agent's configuration: task information, reasoning, self-verification, time budget, and backbone model. We find substantial run-to-run variability, with approximately 54% of the outcome variance coming from repeating the same configuration rather than changing it. Across configurations, the information provided to the agent has the largest effect, exceeding both time budget and model size, while also reducing cost and improving calibration. Configuration choices also interact: additional time helps only when the agent has sufficient information or a capable enough model to use it. Finally, a trajectory-based taxonomy of agent behavior reveals that prompting an agent to verify its answer has little effect on its verification behavior, whereas providing a dedicated verification tool changes that behavior substantially. These results suggest that agents should be evaluated as configurable systems themselves, and that some desired behaviors are more effectively implemented in the system than requested through prompting. We release the benchmark and more than 18,000 agent trajectories.

View source

Similar papers

CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time

This work defines real-time as five measurable properties, each with an equation and a worked example: continuous task arrival, live concurrent execution, online rating updates, freshness with contamination resistance, and bounded feedback latency from a failed run to a reusable training environment.

Nitish Kovuru, Prateek Jannu · 0 citations
#natural language process... Preprint Oct 2026

Scaling Inference-Time Compute with Teams of Coding Agents

This work builds a minimal multi-agent harness from the ground up and traces this frontier by varying, one at a time, the workflow stage at which a lead delegates to its workers (planning, implementation, or verification), the models that form the team, and the number of workers.

Keenan Samway, Daniel D'Souza, O. D. Domingues et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably...

Michael Hardy, Ruhana Azam, Anka Reuel et al. · 2 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.