Skip to content

A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever

Jul 2026 · arXiv.org · Vol abs/2607.23806 · 0 citations · 29 references
Computer Science

TL;DR

On published benchmarks, frontier models remain far ahead of any 12B at raw from-scratch reasoning; on everything this system has solved and verified, the comparison inverts: a frontier API call pays a fresh generation pass on every query, forever, while verified reuse costs zero tokens and returns the identical bits every time.

Abstract

Improving a language model today means retraining it: enormous compute, a new opaque model each cycle, non-deterministic output. We take the opposite path: the model stays frozen, and a persistent memory of verified solutions grows beside it. Once a problem family is solved and has passed an independent verification step that never consults the answer key, every new instance of that family is answered at zero generation tokens, bit-exact, deterministically. Across 180 fresh instances spanning nine problem families, four architectures from four vendors - dense and mixture-of-experts - each score 180/180 at zero generation tokens per answer: execution-bound capability decoupled from parameter scaling. A negative control attributes the capability fully to the memory: emptied, it solves nothing. The same verify-before-store contract holds for open-ended reasoning: 88/88 consistency-gated acceptances across all four models, machine-checked formal proof, and reasoning-method transfer at 77/80. Memory selection takes 1.4 microseconds; a full reuse completes in 6-23 ms at 36 mWh. Approximate similarity retrieval selects the wrong item 94.3% of the time on a 4,500-item verified store where exact addressing makes zero errors. The store also serves as working context at a scale no shipped engine matches: a 6,000,000-token movable window on a single 46 GB GPU at flat memory, where vLLM stops at 30,399 tokens and SGLang silently truncates past 32,000. On published benchmarks, frontier models remain far ahead of any 12B at raw from-scratch reasoning; on everything this system has solved and verified, the comparison inverts: a frontier API call pays a fresh generation pass on every query, forever, while verified reuse costs zero tokens and returns the identical bits every time. A public testbench with free, rate-limited access accompanies this report: https://corbenic-galahad-bench.hf.space

View source

Similar papers

Preprint Aug 2026

Runtime Observability for Heterogeneous Attention Memory

A runtime observability contract is given that covers all four memory classes with three operators, instantiate it on six model configurations across five architecture families, and compose the per-stage bounds into an executable request-level risk ledger.

Fanzhe Wei, Li Liu, Ziyang Wang et al. · 2 citations
Jul 2026

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

A two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness is introduced, demonstrating that accuracy conflates execution case mix with verification policy.

Zong-You Yang, Ying-Han Hou · 0 citations
Preprint Aug 2026

Memory-Sovereign Inference: Output-Exact Execution Beyond Full Residency

A falsifiable certificate is presented separating representation, semantic demand, scheduler requests, and traffic while naming resource authorities, exactness horizons, and tested reuse transitions while naming resource authorities, exactness horizons, and tested reuse transitions.

L. Štěpánek · 0 citations
Jul 2026

IFHierBench: Hierarchical Instruction Following for Large Language Models

IFHierBench is introduced, a hierarchical instruction-following benchmark of 600 prompts stratified across four constraint-tree depths and 35 distinct constraints, each prompt paired with a deterministic checker that verifies satisfaction at every scope.

Yuetian Mao, Chunyang Chen · 0 citations
#artificial intelligence Preprint Aug 2026

Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification

Evaluating state-of-the-art open LLMs reveals a significant robustness gap, and shows that reliable process-level verification remains challenging, and evaluator robustness should be measured separately from solver accuracy, even in a simple algebraic domain with exact ground truth.

Fatemeh Mazdarani, Carlos Toxtli · 2 citations
Preprint Aug 2026

From Test-Time Scaling to Reusable Memory: Measuring Crystallization in Text-to-SQL

A controlled evaluation holds the single-shot solver fixed and varies one memory choice at a time, and separately measures replay, cross-question retention, and held-out same-database transfer in text-to-SQL memory crystallization.

Jia-Qian Wang, Yu-Tao Qi, Wen-Qi Hou et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.