Dual-Axis Benchmark and Preventive Scaffold: Surfacing and Mitigating Silent Risks in LLM Data Preparation
Abstract
Large language model (LLM) agents working on raw, heterogeneous data may fail silently when they act on upstream assumptions that turn out to be wrong. Existing evaluations rarely surface this risk, benchmarking on clean, pre-identified inputs; agentic frameworks respond by layering self-critique or multi-agent debate above the executor. We introduce DiscRun-Bench, a benchmark whose family/variant design decomposes hazard discovery from hazard-aware execution and scores data correctness separately from caveat-surfacing diligence. We then treat PRVR (Plan-Run-Verify-Resolve), a scaffold imitating industry-standard plan-execute-verify scaffold, as a measurement probe. Structuring Claude Haiku 4.5’s operating loop to ask about hazards lifts its caveat-surfacing rate from around 0.74 to 0.90 and its mean task score by roughly 25%, both significant under paired permutation tests. Attribution assigns the gain to the preventive Plan stage and to delivery-layer robustness, not to the reactive self-critique retry. PRVR-Haiku meets or exceeds a bare Sonnet 4.6 reference at roughly 4× lower per-task list-price cost, decomposing into a 3× per-token price ratio and a roughly 1.35× grid-mean token ratio driven by a few high-token cells. Silent failure in LLM data preparation is thus substantially an elicitation problem rather than only a capability problem.