Skip to content
Preprint

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Oct 2026 · 1 citation · 30 references
Computer Science

TL;DR

VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks is introduced; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites.

Abstract

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.

View source

Similar papers

Preprint Nov 2026

Veritas: A Verifier Generation Harness for Agent Tasks

The effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round. However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. We propose Veritas, a harness that generates ver...

Dao-An Zhang, Qin-Bo Bai, De-Hai Min et al. · 0 citations

SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agents

This work introduces SINGED (Source Integrity and the Nonidentifiability Gap in Execution Decisions for LLM Agents), a controlled benchmark covering five primary and two held-out task families and exposes a rank-, evidence-, and choice-sensitive outcome-to-execution gap.

XiaoYu Xu, Zi Liang, Min-Xin Du et al. · 0 citations
#artificial intelligence Preprint Nov 2026

SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification

SEVERUS, a context-aware technique to generate Verus benchmarks that organizes verification tasks into quests, each divided into levels in a gamified manner, provides a foundation for constructing verification benchmarks that can continue to challenge increasingly capable coding agents.

Álvaro F. Silva, Martim Afonso, Nuno Saavedra et al. · 0 citations
Book Open access 2027

From Specification to Kernel Commit: Verified Code Generation on Real-World Systems

Verified code generation, in which an LLM produces both an implementation and a correctness proof, offers a promising path toward correct-by-construction software. Prior evaluations report that LLM capability remains unsatisfactory, but they test models in a bare-model setting that no longer reflects how LLMs are deplo...

Hao Sun, Ze-Nan Li, Cong Li et al. · 0 citations
Preprint Oct 2026

VeriEvolve: Evaluator-First Evolution for Tool-Using Agents

A central obstacle to self-improving agents is the absence of a reliable fitness signal for realistic stateful tasks. We introduce \textsc{VeriEvolve}, an evaluator-first evolution framework that constructs this missing evaluation layer before optimizing the agent. From task instruction, VeriEvolve synthesizes instruct...

Yue-Shen Li, Kamer Ali Yüksel, Gokhan Tur et al. · 0 citations
Preprint Oct 2026

V$^3$AST-Bench: Verifiable, Valid, and Versatile Safety Evaluation for Coding Agents

Across model-harness combinations, agents with comparable competence exhibit markedly different safety outcomes, showing that task capability alone does not guarantee safe execution, and highlighting the need to move beyond risk recognition toward enforcing safety throughout execution.

Wei-Tao Feng, Heng-Heng Xiong, Zi-Hao Zeng et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.