Skip to content
Log in
Conference Open access

Your Reasoning Model is Secretly a Reward Model - Optimization-Free Verification from Experience

Zhenwen Liang Ruosen Li Yujun Zhou Linfeng Song Dian Yu Xinya Du Haitao Mi Dong Yu
2026 · Annual Meeting of the Association for Computational Linguistics · pp. 17358-17372 · 0 citations · 30 references
Computer Science

TL;DR

This paper introduces C LUE (Clustering and Experience-based Verification) , a training-free, non-parametric verifier that improves selection and reranking in Large Language Model outputs and finds that correct and incorrect solutions exhibit measurable geometric differences in their hidden-state trajectories.

Abstract

Assessing the quality of Large Language Model (LLM) outputs becomes especially challenging in high-branching settings, where a single prompt yields many plausible candidates. Existing verifiers typically operate on the surface text (e.g., reward models, LLM judges, majority voting) or on confidence proxies derived from token probabilities, both of which can be brittle: the former can be influenced by stylistic artifacts, while the latter is often miscalibrated. In this paper, we study a third source of information—the model’s hidden states—for binary correctness verification in tasks with a reliable success/failure signal (e.g., deterministic checkers or reference-grounded answers). We find that correct and incorrect solutions exhibit measurable geometric differences in their hidden-state trajectories. To isolate this signal with minimal modeling assumptions, we introduce C LUE (Clustering and Experience-based Verification) , a training-free, non-parametric verifier. C LUE summarizes each reasoning trace by an activation delta —the difference between hidden states at the start and end of the explicit reasoning span— and predicts correctness by comparing this delta to two class centroids computed from labeled experience. Across math (AIME 24/25), scientific QA (GPQA), and a multi-domain benchmark (WebInstruct-verified), C LUE improves selection and reranking

DOI Semantic Scholar
Read PDF

Similar papers

Conference Open access 2026

When Models Hesitate: Answer Instability as a Label-Free Uncertainty Signal for LLMs

This work proposes answer instability, defined as the variability of a model’s final answer across repeated stochastic generations of the same prompt, as a simple, label-free, and black-box uncertainty signal, and demonstrates its utility for selective prediction and targeted repair, improving reliability without access to internal probabilities or additional training.

J. M. Araña, Kristine Ann M. Carandang, Ethan Robert A. Casin et al. · 0 citations

Scaling Test-Time Compute via Generative Verification in Constrained Parameter Regimes

This project investigates scaling test-time compute through a Generative Verifier (GV) on the Countdown mathematical reasoning task using a computationally constrained 0.5B parameter regime, hypothesizing that the “verification gap” will widen at higher values of N due to the model’s limited semantic capacity.

Mohammad Rehan Ghori · 0 citations
Preprint Aug 2026

AutoVerifier: Residual-Guided Non-Parametric Optimization for Reference-Based Answer Verification

Reference-based verifiers are important for evaluating reasoning models and providing accurate outcome rewards in reinforcement learning with verifiable rewards. To improve verification accuracy, prior work has explored rule-based, model-based, and tool-augmented verifiers for checking answer equivalence across diverse answer forms. However, the equivalence of answer forms such as $1+3.14$ and $1+\pi$ may depend on the question and scoring criterion. We frame such implicit assumptions as verifier inductive biases. To address this challenge, we propose AutoVerifier, a residual-guided non-parametric optimization method that learns these biases from recurring verifier errors. Specifically, AutoVerifier records these biases in rule cards and promotes them to code modules or prompt guidance only after replay validation detects no direct regressions, keeping accepted updates auditable, editable, and reusable. Experiments on four verifier benchmarks demonstrate that AutoVerifier outperforms state-of-the-art verifiers by a large margin.

Zelong Zhao, Zhihui Shi, Min-Qi Shi · 0 citations
Preprint Aug 2026

When Many Answers Are Valid, Voting Fails: Symbolic Verification for Best-of-K Causal Reasoning in LLMs

This work introduces CALVER (Causal Axiom-Level VERification), a training-free symbolic verifier that scores structured traces against Pearl's causal criteria, including -separation, backdoor adjustment, and intervention, and selects the highest-scoring candidate without consulting a reference answer.

Omatharv Bharat Vaidya, C. T. Jerzak, Zayne Sprague et al. · 0 citations