Skip to content

Pipelining the Memory Hierarchy for Ultra-Long-Context Language Model Training

Nov 2026 · GetMobile Mobile Computing and Communications · Vol 30, pp. 5 - 8 · 0 citations · 7 references

TL;DR

FPDT democratizes long sequence training, making it accessible on mobile and commodity hardware, such as high-end local machines, and trains contexts up to 16x longer than prior sequence-parallel baselines on the same hardware.

Abstract

Long-context language models are attractive because they can read whole documents, code repositories, conversations, and scientific sequences. However, a long inference window is not the same as a model that was trained to use millions of tokens reliably. Direct long-context training is blocked by temporary memory spikes from activations, attention buffers, communication workspaces, and the final loss computation, even when model parameters and optimizer states are already sharded. Our proposed method, Fully Pipelined Distributed Transformer (FPDT), treats the long sequence problem as a memory-hierarchy scheduling problem. It chunks each sequence before distributed attention, stages inactive query/ key/value tensors in host memory and uses double buffers to overlap transfers with attention computation. FPDT democratizes long sequence training, making it accessible on mobile and commodity hardware, such as high-end local machines. In our evaluation, FPDT trains contexts up to 16x longer than prior sequence-parallel baselines on the same hardware. It reaches a 2M-token context for an 8B model on four A100-80GB GPUs, and a 4M-token context for 8B Llama 3 on eight A100-80GB GPUs while sustaining 55.7% Model FLOPs Utilization (MFU) [1].

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Hybrid Latent Attention for Looped Language Models

Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key-value (KV) cache by T. The larger cache limits how many sequences a GPU can decode at once and slows each decoding step, which reads the whole cache. We propose Hybrid La...

Yu-Han Chen, Si-Yuan Zhang, Nan Wang et al. · 4 citations · ⚡1

From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

Long-context capability has become essential for large language models (LLMs) to handle complex tasks, yet continuously expanding the context window does not remove the computational, storage, and information-selection pressures incurred at inference time. Existing work on prompt compression, KV-cache management, effic...

Chao-Gui Gou · 0 citations
#machine learning Preprint Oct 2026

FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

This work proposes FOLD, a training-free method that preserves exact local positions and treats all distant entries as far memory at a shared position and sharpen attention within it to emphasize relevant entries while separately controlling its total weight relative to the local window.

Shu-Xuan Li, Jia-Liang Kang, Yang-Hao Zhou et al. · 0 citations
Preprint Nov 2026

Shallow Encoding Emits Deep Memories

SEED makes the prefill shallow in depth and the decode state shallow in representation, and proves that the open-loop error of the periodic compressions back to low rank is bounded independently of their number.

Ming-Yuan Ma, Yang-Min Li, H. Nover et al. · 0 citations
#machine learning Preprint Oct 2026

Universal Test-Time Training

Recent Test-Time Training (TTT) architectures compress context into fast weights that are updated online and queried as memory. Existing TTT designs keep this memory private to each layer: it recurs only over time, and depth merely indexes L separate memories. We argue that memory ownership need not be tied to depth, a...

Ze-Fan Cai, Qin-Zhe Hu, Ziqiao Ma et al. · 1 citation
Preprint Nov 2026

LiquidCP: Decoupling Attention Tasks from Communication and Kernel Boundaries in Context Parallelism

Context parallelism (CP) partitions long sequences across GPUs for LLM training and inference. For variable-length training and full prefill, attention-balanced placement can leave token loads uneven, while ring execution repeats overhead at each computation and communication step. In stage-serialized decode CP (DCP),...

Hong-Yu Chen, Yi-Kang Ruan, Yu-Hai Long et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.