This work introduces EigenLI, a spectral approximation framework that compresses late-interaction representations via document-specific low-dimensional subspaces and consistently outperforms comparable single-vector surrogates such as MUVERA.
Abstract
Late-interaction models such as ColBERT achieve strong effectiveness by representing each document with many token-level vectors, but this expressivity leads to large indexing cost, storage footprints and expensive MaxSim scoring. We show that late-interaction representations exhibit an intrinsic low-rank structure: document token embeddings concentrate in a low-dimensional subspace that preserves most of the retrieval signal. Leveraging this observation, we introduce EigenLI, a spectral approximation framework that compresses late-interaction representations via document-specific low-dimensional subspaces. Unlike clustering or pooling methods, EigenLI identifies the dominant eigendirections of each document and uses them to construct reduced interaction representations. Empirically, $k$-EigenLI with $k \le 32$ outperforms k-means and Ward clustering based pooling methods on ColBERTv2 and AnswerAI-ColBERT-small; GTE-ModernColBERT exhibits a different tradeoff at $k=32$, where clustering methods perform better. The same spectral construction also yields EigenLI-SV, an ANN-compatible single-vector representation derived from the second-order summary of the reduced structure. Across multiple datasets and all three text models, EigenLI-SV consistently outperforms comparable single-vector surrogates such as MUVERA.
Generative Late-Interaction Embeddings (GLIE): k<<N vectors per page learned from the normalized centroids to serve as both a lightweight index and a basis for regenerating the page's full embedding set, with the decoder as its main design surface.
Mohamed Eltahir, Talal Aloushan, Rose Khairoalsendi et al.· 0 citations
This work provides the first explicit family of query and document sets, together with their relevance matrices, for which single-vector embeddings that rank all relevant documents above irrelevant ones require exponential size, whereas polynomial-size multi-vector embeddings suffice.
Mihir Agarwal, Viraj Agrawal, Sabyasachi Basu et al.· 2 citations
This work introduces MARETopic, a training-free framework that casts topic discovery as rank-based prototype selection and leads topic coherence on two of three datasets, ahead of both neural and clustering-based topic models.
Thiago César Castilho Almeida, D. Pedronette· 0 citations
It is demonstrated that models trained using ColSNAP maintain near full-resolution retrieval performance under substantial compression and that ColSNAP transfers effectively across multiple late-interaction backbones, and achieves most of its improvements via a lightweight adaptation stage applied to a pre-trained retr...
T. Roy, Arkadeep Acharya, Vishwajeet Kumar et al.· 1 citation
AdaMerge is proposed, a plug-and-play compression method that detects each document's own cliff via gap analysis on the merge-cosine trajectory, and builds attention-weighted cluster centroids to preserve salient signal.
J.-H. You, Kun Ni· Proceedings of the 35th ACM...· 0 citations
It is argued that effective compression should preserve query-relevant coverage, meaning the diverse document regions that may become the strongest MaxSim match across queries, rather than selecting patches independently by salience, why dense rendered pages are easier to compress than natural images.
Ailar Mahdizadeh, Aria Salari, Sohail Rajabi et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026