Skip to content

The Topological Trouble With Transformers

Apr 2026 · arXiv.org · Vol abs/2604.17121 · 3 citations · 101 references
Computer Science

TL;DR

It is argued that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures via recurrent architectures, and a taxonomy of recurrent and continuous-thought transformer architectures is introduced.

Abstract

Transformers encode structure in sequences via an expanding contextual history. However, their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking -- the iterative updating of latent variables reflecting an evolving environment -- involves inherently sequential dependencies that feedforward networks struggle to maintain. Consequently, feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model's depth. While this depth limit can be bypassed by dynamic depth models and by explicit or latent thinking that externalizes state representations, these solutions are computationally and memory inefficient. In this article, we argue that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures. We introduce a taxonomy of recurrent and continuous-thought transformer architectures, categorizing them by their recurrence axis (depth versus step) and their ratio of input tokens to recurrence steps. Finally, we outline promising research directions, including enhanced state-space models and coarse-grained recurrence, to better integrate state tracking into modern foundation models.

View source

Similar papers

#machine learning Preprint Aug 2026

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform...

Amr Hegazy, Amr Alanwar, Mostafa Elhoushi · 2 citations
#machine learning Preprint Aug 2026

Dynamic Compression in Recurrent Networks

Recurrent models process long contexts efficiently by compressing their history into a fixed-size state, but modern architectures typically do so in a single causal pass over the sequence. Each input must therefore be compressed before the model knows how it will later be used, forcing a limited state to compromise acr...

Jyothish Pari, Ryan Bahlous-Boldi, Pulkit Agrawal · 0 citations
#artificial intelligence Preprint Sep 2026

A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM

A*-Thought-V2 is presented, a geometric dynamics of LLM guided framework that models CoT as a hidden-state trajectory and replaces hard deletion with an explicit-implicit interleaved latent architecture that reflects broader soft targets that encourage richer step-level feature learning.

Xiao-An Xu, Si-Yuan Liu, Shuo Wang et al. · 1 citation

Scheduling Recursive Reasoning in Looped Transformers

Recurrent reasoning models have attracted growing attention for scaling test-time computation, typically by iteratively refining latent states with shared parameters. However, these models apply each learned update with a fixed unit scale, which can be conservative when updates make persistent progress and overly aggre...

Bo-Yuan Wang, Cheng-Yao Yu, Jia-Xiang Ren et al. · 1 citation · ⚡1
Preprint Aug 2026

Allocating Recurrent Compute in Looped Language Models

Results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN, and view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout.

Ruhai Lin, Yi-Yang Guo, Ruiming Zhu et al. · 4 citations
#machine learning Preprint Sep 2026

Pattern Formation in Transformers

What are the inductive biases of a Transformer architecture? Existing theory on how the forward pass shapes representations either considers whether Transformers escape from rank collapse or demonstrates that self-attention drives tokens toward cluster patterns. The latter view arises from an elegant dynamical systems...

Erkan Turan, Gaspard Abel, M. Ovsjanikov · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.