Um, Actually: Timely and Calibrated Fact-Checking with DIVER

Abstract

Fact verification pipelines are often unrealistic: they assume full-input access, operate on isolated atomic claims, and overlook the central question of when a system should act under partial evidence. We introduce DIVER (Deliberative and Iterative fact VERification), an LLM -based framework for calibrated incremental fact-checking. DIVER incrementally extracts verifiable atomic claims from progressively revealed input, retrieves and refines evidence through iterative web search, and triggers a correction as soon as a claim is reliably refuted; if no such trigger occurs, it performs a passage-level revision over the full input to recover missed errors. Across standard fact-checking benchmarks, DIVER outperforms strong LLM -based verification pipelines, with especially large gains on retrieval-intensive settings. We further introduce U M A CTUALLY , a real-world benchmark built from the Internet game Um, Actually , to evaluate not only whether a system finds an error, but whether it does so at the right time. On U M A CTUALLY , DIVER reports the strongest end-to-end results while typically buzzing close to the earliest valid decision point, supporting its effectiveness for calibrated incremental fact-checking. 1