Skip to content

What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetition

Aug 2026 · 0 citations · 10 references
Computer Science

TL;DR

An observational measurement in a different currency is reported: the fixed-point structure of a model's own short-window argmax map, censused from 96 random two-token starts over 17 off-the-shelf models, always unprompted.

Abstract

Two accounts of neural text degeneration coexist. One locates the cause in the training data -- repetition in the corpus produces repetition in the output, established by training on repetition-sorted data -- the other in the trained network, in copying circuits and repetition features. Neither has been arbitrated across a broad cohort of pretrained models: the causal work trains its own. We report an observational measurement in a different currency: the fixed-point structure of a model's own short-window argmax map, censused from 96 random two-token starts over 17 off-the-shelf models, always unprompted -- a companion paper shows nine tokens of conditioning move this readout across most of its range. The four-way class is stable across census seeds on 17 of 17. Three exhibits. At fixed corpus (The Pile), fixed scale and that fixed domain, the class is not determined: across two size-matched tiers, pythia is a funnel while RWKV, Mamba and a second transformer family are not, and both hold their class across an order of magnitude of scale. Six of seven models in that ladder reach the same endpoint token, and those concentrating on it most strongly are among those that never stay there -- what varies is not where trajectories go but whether the destination self-continues. The deduplicated Pythia suite does not change the class. And the corpus-side inflow term proposed for this phenomenon does not select our endpoints once frequency is controlled, in English and three other languages. This is observational and cannot refute a training intervention. Funnels are common: eight of seventeen models, seven families, five corpora -- so the limit is not that the phenomenon is one model's peculiarity, but that within the one corpus where training data can be held fixed only one available family funnels; that subset cannot show the split is corpus-independent.

View source

Similar papers

#natural language process... Open access Oct 2026

Copying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?

Mechanistic interpretability usually studies fully trained models, yet the computations that drive a behaviour can change while the model is still learning the task. On the Indirect Object Identification task, a model should continue with the name mentioned once rather than the name mentioned twice. Pythia models pass...

T. Dahiya, Cole Blondin · 1 citation

Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction

This empirical study is an independent reproduction of the dissociation Zhao reported in 2026, where the shape of a large language model's chain-of-thought entropy trajectory predicts whether the final answer is correct, while the magnitude of its total entropy drop does not.

Theodore O. Cochran · 0 citations

What You Can't See Is Still What You Learn: A Preregistered Sixty-Society Confirmation That Evidence Masking Drives Compositional Generalization

A preregistered confirmation with sixty four-cell systems sharing a frozen language-model backbone and communicating through learned continuous packets confirms a large advantage of the tested masking regime, while leaving its finer attribution and generality open.

Narcis Marincat · 0 citations
#artificial intelligence Conference Sep 2026

The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance

Using large language models (LLMs) to simulate diverse human populations has the potential to transform many aspects of computational social science, yet many evaluations score the average response rather than the spread of opinion within real groups. Here, we develop a diagnostic framework that measures point accuracy...

Rojin Ziaei · 1 citation
Preprint Sep 2026

Identifiability of Neologism Dynamics from Aggregate Corpus Counts

Aggregate lexical frequencies reflect both what speakers do and how a corpus records it. We develop a stochastic framework for lexical adoption, abandonment and reactivation on a temporal multilayer network, together with a count observation model separating speaker prevalence, use intensity and detectability, and ask...

Iván Area, Antonio Moreno-Sandoval, Jordi Porta-Zamorano · 0 citations
Preprint Oct 2026

Not All Test-Time Compute Is Evidence: Model-Specific Bias in LLM Reasoning

Test-time scaling methods implicitly assume that additional computation from the same model provides independent evidence. We show that this assumption fails within a single reasoning model: errors are correlated across repeated samples, across reasoning depth, and in the model's own self-evaluation, so more compute fr...

Ming Shen, Tian-Jun Yao · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.