It is observed that, given layer inputs and activation gradients (costates), online gradient descent admits exact parallel scans for both forward evaluation and reverse backpropagation.
Abstract
Test-time training (TTT) allows a model to improve its predictions at inference time by updating weights after every observed token. However, sequential gra- dient writes make parallel training difficult. We observe that, given layer inputs and activation gradients (costates), online gradient descent admits exact parallel scans for both forward evaluation and reverse backpropagation. GradLev lever- ages this duality: a causal auxiliary network predicts costates across all tokens in parallel; associative scans compute the adapted weights and forward activations and propagate gradients backward; and the resulting gradient targets supervise the predictor via a consistency loss. Exact consistency guarantees exact recovery of the sequential online learner. At deployment, the auxiliary predictor is discarded, and the model updates natively via token-by-token forward and backward passes.
Under the standard approximation of taking gradients at the chunk-start weights, a closed-form state transition is derived that exactly reproduces the chunk-end fast-weight and momentum states of the per-token recurrence of Test-Time Training.
Zeyun Zhong, Joya Chen, Manuel Martín et al.· 2 citations
M measuring the cost of a memory cut by training with it from initialization, and comparing backward graphs by the updates the optimizer applies rather than by raw gradients suggest the cost of a memory cut should be measured by training with it from initialization.
Xing-Jian Li, Jian-Hua Z. Huang, Jun-Li Duan· 0 citations
This work intervenes directly on state credit and proposes Credit Stabilization through Time (CST), which locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged.
Test-time training (TTT) lets a model store information in its weights during inference. When the model learns from its own output, however, each update also changes the model that generates the next training example. Across 128K-token streams, retaining generated-text updates worsens prediction on independent human-wr...
Cheng-Zhi Luo, Bing Li, Bernard Ghanem· 0 citations
Backpropagation (BP) dominates deep learning but imposes a massive memory tax. For example, training OPT-30B with Adam requires $\approx$ 600GB of GPU memory (assuming batch size 8 and sequence length 2048). Alternatively, zero-order optimization (ZOO) trains in inference-mode (requiring only $\approx$ 60GB for the sam...
Francois Chaubard, M. Kochenderfer, Chris Ré· 0 citations
This work adds a persistent hidden state to a diffusion denoiser and removes its timestep conditioning, leaving a single shared update that can be run to arbitrary depth, and develops an anytime solver that keeps improving with inference depth far beyond the rollout lengths and backpropagation window used in training.
M. Drozdova, Aidan Sirbu, Pietro Miotti et al.· 3 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.
