SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split...
Soham Dan, Jeff Da, Sanyam Bhutani et al.· 1 citation· ⚡1
BARC-Bench is introduced, a collection of 110 tasks across nine simulated workspace environments where benign requests leave openings for unsafe actions, and four frontier model configurations are evaluated, highlighting an asymmetric operational risk.
Kang-Kang Li, Kelvin Luu, Li Ma et al.· 0 citations
Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the execution trace. A judge may overlook this evidence or accept unsupported claims, including an agent's own reports of success. When rubric v...
Reza Sanayei, Razvan-Gabriel Dumitru, Sun-Jie Hou et al.· 0 citations
A process-verification framework is introduced that audits passing trajectories, distinguishes evidenced reward hacking from verifier weakness, and localizes exploitable surfaces for repair in benchmark validity.
Wei-Jun Luo, Kelvin Luu, Xin-Yi Liu et al.· arXiv.org· 4 citations
Reliable Enterprise Agent Deployment (READY), a framework for qualifying AI agents for deployment on enterprise workflows, and provides a basis for comparing agent systems, setting oversight requirements, and making evidence-based deployment decisions.
Veronica Chatrath, Bryan Zhu, Jing-Xuan Fan et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.