CatchBench: When Can an Agent Failure Be Caught?

Abstract

When can an agent failure be caught? A weak audit score alone cannot identify whether the record or the method is limiting. CatchBench therefore puts one auditor's question to three information states: the declared configuration before a run (PRE), a growing prefix of its trace (LIVE), and the finished trace (POST). Prior benchmarks fix one of these states or vary the telemetry; to our knowledge none scores all three under one task-method interface. Each state admits different questions, so seven task contracts carry their own labels and metrics rather than one leaderboard. Four are evidential; three are Gold-derived mechanism diagnostics. The release scores 72 entrants, from rule scanners and structural models to eleven LLM judges across nine model families (GPT, Claude, Gemini, Gemma, Llama, Qwen, DeepSeek, Mistral, Nova), over 1187 declared configurations and 1162 recorded runs. Every recorded comparison is published as a measured difference with its interval, uncorrected, and no board declares a winner it cannot show. The three sharpest results cut against our own data. One rule reads declaration order alone and reaches a perfect F1 on one of six configuration sources, so a score there measures how the corpus was built. Our admissibility bar then rejected one injected substrate and withheld evidential status from the other. A published structural gain also turns on which size reference it is measured against. A benchmark number is therefore not interpretable until the process behind its labels is published and tested for the shortcut it may leave. We report all three, and regenerate every ordering from released predictions with no model call.