Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

S. Dhruvil Fenil Sojitra R. Chauhan

Jul 2026 · arXiv.org · Vol abs/2607.23054 · 0 citations · 25 references

Computer Science

TL;DR

This work presents the first comprehensive mechanistic interpretability study of MLA, training a 114M-parameter transformer and analyzing its representations through SVD, attention head taxonomy, linear probing, and a disruption-attribution analysis.

Abstract

Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference. Despite its adoption in massive production models, no prior work has studied what information this bottleneck preserves or discards, nor how it reshapes internal transformer circuits. We present the first comprehensive mechanistic interpretability study of MLA, training a 114M-parameter transformer (pretrained on a web/code/math mixture, fine-tuned on TinyStories) and analyzing its representations through SVD, attention head taxonomy, linear probing, and a disruption-attribution analysis. Our key findings are: (1) the cKV bottleneck learns a pure content representation, preserving entity identity (98% retention) while discarding positional information, validating MLA's separation of content from position via RoPE; (2) induction heads co-locate at a single layer (Layer 12), unlike their distributed formation in standard MHA; (3) a single"semantic hub"layer (Layer 15) simultaneously exhibits the highest SVD effective rank and strongest disruption-attribution score; and (4) the bottleneck is globally over-provisioned, using only 46% of its capacity on average. These findings suggest MLA does not merely compress attention passively, but reshapes how the model organizes content, position, and circuit structure. We view this as an initial data point and detail scope limitations in Section 5.

DOI arXiv Semantic Scholar

View source

Similar papers

Jul 2026

Multi-Head Attention Residuals

Multi-Head Attention Residuals (MHAR) is introduced: the routing query is reshaped into H per-subspace heads, each with its own softmax over the depth history, and a direct probe of the trained queries confirms that learned subspace disagreement is the underlying driver.

Cheng Luo, Zefan Cai, Jun-Jie Hu · arXiv.org · 1 citation

#machine learning Preprint Aug 2026

Can LLMs Use Relational Transformer Embeddings?

It is argued that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.

Francisco Galuppo Azevedo, Clarissa Lima Loures · 0 citations

Preprint Aug 2026

LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

This work argues that cross-modal alignment is implicitly captured in the information-compression trajectory, and proposes LLaVAFlow, an information-theoretic distillation framework that preserves alignment flow and enhances both downstream performance and generalization.

Mu-Yao Yuan, Mu-Yan Jiao, Jiang-Yong Ying et al. · 0 citations

Preprint Aug 2026

Does the LM Head Create a Harmful Gradient Bottleneck? A Causal Test

Controls show that repeated-token failures are confounded by the number of independently sampled symbols, that adding never-target output classes does not impair learning, and that projection diagnostics do not reliably predict progress in the authors' runs.

A. Murugan · 1 citation

Jul 2026

Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models

B1ade, an efficient RAG architecture comprising two purpose-built components: a compact embedding model and a purpose-built SLM shows that strategic model composition and reward design suffice for resource-efficient RAG, without large-scale pretraining.

S. Subramanian, M. Gungor, Vikram Elango · arXiv.org · 1 citation

Preprint Aug 2026

LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining

Large language models (LLMs) have achieved remarkable breakthroughs across various applications. However, their architectures remain inefficient in pretraining due to two main limitations: (i) self-attention lacks an explicit inductive bias for locality, leading to redundant modeling of sequence-internal local informat...

Qiuwu Chen, Zi-Mo Liu, Yuchen Li et al. · 0 citations