Skip to content

Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence

Sep 2026 · arXiv.org · Vol abs/2609.39082 · 0 citations · 30 references
Computer Science

TL;DR

It is shown that high-dimensional spectral memory does not require high-dimensional control, and two shared control signals can efficiently govern adaptive spectral memory across online and full-sequence settings, and Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence (SPARC) is introduced.

Abstract

As new evidence arrives, a sequence model must update what it remembers and how memory influences predictions. While Transformers incur computation and cache costs scaling with context length, fixed-state recurrent models offer constant-memory inference. However, linear and spectral recurrences traditionally rely on static transitions, failing to dynamically revise how stored representations decay or rotate. While recent selective architectures introduce input-dependent transitions, they assign independent controls to every memory mode, coupling control cost to state capacity. We show that high-dimensional spectral memory does not require high-dimensional control, and introduce Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence (SPARC). SPARC employs just two input-dependent scalar signals to coordinate memory retention and phase rotation across heterogeneous complex modes, while preserving mode-specific baseline timescales and frequencies. Its diagonal affine recurrence supports parallel associative scans for sequence-level BPTT as well as exact structured Real-Time Recurrent Learning (RTRL) for online credit assignment. Across partially observable continuous control, POPGym, and sequence classification, SPARC achieves a 9.09% relative return improvement on Walker-P and a 1.36% relative accuracy gain on FordA over second-best methods. On an NVIDIA Blackwell GPU, our implementation reduces recurrent-mixer training latency by 18.2%-34.2% in fixed-token workloads and accelerates scans by 3.1x-4.7x over an optimized RG-LRU baseline. These results show that two shared control signals can efficiently govern adaptive spectral memory across online and full-sequence settings. Code is available at https://github.com/Botwwt/sparc.

View source

Similar papers

CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory

Linear RNNs offer linear-time sequence processing and constant-memory decoding, but their fixed-size recurrent states must accommodate all past key--value associations. Existing forgetting mechanisms and Delta Rule updates reduce interference by selectively clearing or correcting the state, yet earlier associations can...

Yi-Xiao Chen, Shuo-Jin Yang, Shi-Min Hu · 0 citations

How Linear Attention Remembers

Together, these results reveal how fixed-size recurrent memory supports selective recall despite shared storage, while exposing the interference and capacity limits that distinguish it from token-addressable KV memory.

Kichang Lee, JaeYeon Park, Songkuk Kim et al. · 0 citations

Elastic Selective Spectral Hybrids for Train-Once, Export-Many Budgeted Inference

The Elastic Selective Spectral Hybrid is introduced, which realizes each Hankel spectral channel as an independent recurrent unit using fitted damped rotation modes and achieves language-modeling quality comparable to similarly sized independently trained models, while smaller exports exhibit a smooth quality-cost trad...

Da-Chuan Song, Chu-Chu Chen, Xuan Wang · 0 citations
#artificial intelligence Preprint Oct 2026

Continuous Memory Machines

Recurrent neural networks typically compress information into a single vector-valued recurrent state, forcing short-term computation and long-term retention to share the same representation. Past extensions alleviate this bottleneck by increasing the memory capacity or separating timescales, but lack the combination of...

Ciaran Regan, K. Arulkumaran, L. Darlow et al. · 0 citations

RPMem: Learning Long-Term Recurrent Parametric Memory Across Sessions for LLM Agents

RPMem is introduced, a two-stage architecture that compiles each session into a model-independent latent memory through forward computation and selectively integrates it with retained memory via a task-trained recurrent gate, establishing RPMem as a lifecycle-independent parametric memory framework that maintains evolv...

Fan-Yu Zhao, Rui-Ke Cao, Liang Dong et al. · 1 citation · ⚡1
#artificial intelligence Preprint Oct 2026

DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention

Linear attention scales linearly with sequence length, but its finite-state memory can limit long-context retrieval. In gated linear recurrences, memory retention depends on state decay, which is shaped jointly by gate parameters A_log and dt_bias and by input activations. In controlled Gated DeltaNet (GDN) experiments...

Zhang-Chi Feng, Harry Jiang, Yangyifei Luo et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.