An observational measurement in a different currency is reported: the fixed-point structure of a model's own short-window argmax map, censused from 96 random two-token starts over 17 off-the-shelf models, always unprompted.
Abstract
Two accounts of neural text degeneration coexist. One locates the cause in the training data -- repetition in the corpus produces repetition in the output, established by training on repetition-sorted data -- the other in the trained network, in copying circuits and repetition features. Neither has been arbitrated across a broad cohort of pretrained models: the causal work trains its own. We report an observational measurement in a different currency: the fixed-point structure of a model's own short-window argmax map, censused from 96 random two-token starts over 17 off-the-shelf models, always unprompted -- a companion paper shows nine tokens of conditioning move this readout across most of its range. The four-way class is stable across census seeds on 17 of 17. Three exhibits. At fixed corpus (The Pile), fixed scale and that fixed domain, the class is not determined: across two size-matched tiers, pythia is a funnel while RWKV, Mamba and a second transformer family are not, and both hold their class across an order of magnitude of scale. Six of seven models in that ladder reach the same endpoint token, and those concentrating on it most strongly are among those that never stay there -- what varies is not where trajectories go but whether the destination self-continues. The deduplicated Pythia suite does not change the class. And the corpus-side inflow term proposed for this phenomenon does not select our endpoints once frequency is controlled, in English and three other languages. This is observational and cannot refute a training intervention. Funnels are common: eight of seventeen models, seven families, five corpora -- so the limit is not that the phenomenon is one model's peculiarity, but that within the one corpus where training data can be held fixed only one available family funnels; that subset cannot show the split is corpus-independent.
Mechanistic interpretability usually studies fully trained models, yet the computations that drive a behaviour can change while the model is still learning the task. On the Indirect Object Identification task, a model should continue with the name mentioned once rather than the name mentioned twice. Pythia models pass...
T. Dahiya, Cole Blondin· Conference on Empirical Meth...· 1 citation
This empirical study is an independent reproduction of the dissociation Zhao reported in 2026, where the shape of a large language model's chain-of-thought entropy trajectory predicts whether the final answer is correct, while the magnitude of its total entropy drop does not.
A preregistered confirmation with sixty four-cell systems sharing a frozen language-model backbone and communicating through learned continuous packets confirms a large advantage of the tested masking regime, while leaving its finer attribution and generality open.
Using large language models (LLMs) to simulate diverse human populations has the potential to transform many aspects of computational social science, yet many evaluations score the average response rather than the spread of opinion within real groups. Here, we develop a diagnostic framework that measures point accuracy...
Aggregate lexical frequencies reflect both what speakers do and how a corpus records it. We develop a stochastic framework for lexical adoption, abandonment and reactivation on a temporal multilayer network, together with a count observation model separating speaker prevalence, use intensity and detectability, and ask...
Iván Area, Antonio Moreno-Sandoval, Jordi Porta-Zamorano· 0 citations
Test-time scaling methods implicitly assume that additional computation from the same model provides independent evidence. We show that this assumption fails within a single reasoning model: errors are correlated across repeated samples, across reasoning depth, and in the model's own self-evaluation, so more compute fr...
Ming Shen, Tian-Jun Yao· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.