As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenari...
Zhiqing Zhong, Ming-Yu Chen, Ao-Yang Fang et al.· 0 citations
The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test pass-rate is a natural alternative. However, we find that it is not a reliable indicator of patch quality. A functionality being tested can...
Junjielong Xu, Hua-Chao Zhu, Alyssa Wu et al.· 0 citations
As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users often begin with vague intent, leaving important requirements to be clarified through interaction. In repository-level development, misunderstandings can propagate and cause...
Ming-Yu Chen, Zhiqing Zhong, Man-Yi Wang et al.· 0 citations
PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks, adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection.
