Nikolai Ludwig

Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks

This work systematize and audit exploits across five open large language models on SWE-bench Multilingual and DeepSWE using a turn-level LLM-as-a-judge protocol, demonstrating the critical need for exploit-aware evaluation frameworks that measure true repository-level problem solving over benchmark gaming.