TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications
TestSuiteBench is introduced, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants, showing agents can build software they cannot verify, a gap that directly limits fully autonom...
Kazuki Egashira, Niels Mündler-Sasahara, Mark Niklas Müller et al.
· 0 citations