Jul 2026
Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction. Their scores support capability claims only when the evaluation protocol keeps the intended capability necessary for success. Recent reward-hacking benchmarks and system reports show that agents can inst...
Jia-Qi Shao, Hanck Chen, Wei Zhang et al.
· arXiv.org · 4 citations