Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference
Sparse long-context inference requires efficient token retrieval in both prefill and decode. Existing methods often use different retrieval strategies for the two stages, preventing one retrieval representation from being reused throughout inference. We propose Self-Indexing Attention, a training-free framework built o...
Xu Yang, Jia-Peng Zhang, Zhangke et al.
· 0 citations