Modern coding agents rely on establishing executable software environments before they can perform software engineering tasks such as testing, debugging, and patch validation. Existing setup agents typically control progress at either single-command or whole-script granularity, making recovery either too local to remov...
Xiang Li, Si-Yu Lu, Federica Sarro et al.· 0 citations
Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks. We introduce Zero2Repo, a benchmark in which an agent receives a product requirements document, an interface...
Pei Yang, Tian-Yu Shi, Yu-Hang Yao et al.· 0 citations
Harbor Adapters are developed that port more than 80 benchmarks to evaluate arbitrary agents, and validated through rigorous code review and parity experiments, and Harbor-Index is introduced, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks to support more reliable and comprehensiv...
Lin Shi, Hao-Wei Lin, Zi-Xuan Zhu et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.