A software feature usually evolves over time: developers extend its first version as new use cases and requirements arrive, and each extension builds on code written earlier. Most feature benchmarks evaluate a single request on a fixed codebase. Feature evolution requires evaluating successive extensions on an agent's...
Yun-Long Wang, Alyssa Wu, Cui-Yun Gao et al.· 0 citations
The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test pass-rate is a natural alternative. However, we find that it is not a reliable indicator of patch quality. A functionality being tested can...
Junjielong Xu, Hua-Chao Zhu, Alyssa Wu et al.· 0 citations
This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting and proposes a Checklist-Based Verification Protocol that decomposes answers into atomic,...
Yue Yang, Alyssa Wu, Ji Luo et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.