Skip to content
Open access

Characterizing the role of reasoning tokens in large language models as state over tokens

Oct 2026 · Nature Machine Intelligence · Vol 8, pp. 1515 - 1522 · 1 citation · 69 references

TL;DR

The State over Tokens (SoT) conceptual framework is introduced, which reframes reasoning tokens not as a linguistic narrative, but as an externalized computational state—the mechanism that persists the computation across the separate generation cycles.

Abstract

Large language models increasingly rely on generating reasoning tokens that appear to describe a reasoning process performed by the model to arrive at the answer. However, empirical evidence reveals that when read as text, these tokens are not a faithful explanation of the model’s actual reasoning process. To address this gap between appearance and function, we introduce the State over Tokens (SoT) conceptual framework. SoT reframes reasoning tokens not as a linguistic narrative, but as an externalized computational state—the mechanism that persists the computation across the separate generation cycles. The SoT framing explains how the tokens can drive correct reasoning without being a faithful explanation when read as text and surfaces previously overlooked research questions on these tokens. We discuss how it dispels two common misconceptions: that the text records the complete process, and that the model uses the words with the meanings a human reader ascribes to them. We argue that this framing offers a fruitful way to think about these tokens, one that captures their mechanics accurately while remaining simple and intuitive, and that adopting it opens research directions. To better understand the mechanism of large-language-model reasoning, research should move beyond reading the tokens as text and also focus on decoding them as state. When large language models ‘reason’ about a problem, they might still solve it differently from what their reasoning transcript suggest. In this Perspective, Levy et al. describe how one can start investigating the role of these reasoning tokens.

Read PDF

Similar papers

#artificial intelligence Open access Oct 2026

Talking with language models

The artifactual stance is introduced, which describes human–AI interaction in terms of their design and affordances, rather than potential intentional attitudes, and reframes puzzles about AI reference, the persistence of supposed interlocutors, and responsibility for AI-generated text.

James Ravi Kirkpatrick, Alexandru Radulescu, R. Sterken · 0 citations
Open access

Reasoning in Language Models

The work done in this thesis indicates that LLMs do exhibit properties and behaviors one would expect of a system capable of reasoning, but always with caveats: models solve tasks requiring reasoning only some of the time, and the relations between concepts and propositions appear to be reflected in their internal repr...

Stefan F. Schouten · 5 citations

Chronologic: Measuring Language Models' Ability to Represent the Past

Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its responses fit the period represented. Validation is challenging, because this is not a task living people ordinarily perform, and because many questions have multiple correct an...

T. Underwood, Zi-Liang Qiu, Sarah Griebel et al. · 1 citation
#natural language process... Open access Oct 2026

Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation

This work builds a fully monolingual German RAG question-answering testbed over the fictional world of the tabletop role-playing game The Dark Eye, a domain that is richly documented in German but too niche for the model to answer from memory, so that it has to rely on retrieval.

Oliver Hauck, Mario Sanz-Guerrero, Katharina von der Wense · 0 citations
#natural language process... Conference Open access Sep 2026

From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Models

A declarative probabilistic framework for pre-training evaluation that makes the semantic structure of model behavior explicit and provides new formal tools for relating evaluation to learning, and links evaluation and learning through a shared semantics.

Kyle Richardson, C. Anderson, Pranav Balakrishnan et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.