Subquadratic Attention with a Hierarchical Tree-Structured Indexer
This paper derives an asymptotically subquadratic sparse attention method based on a hierarchical, tree-structured indexer, which it is demonstrated can be cheaply applied to existing pretrained models, and maintains performance on long-context retrieval and challenging reasoning, mathematical, and agentic coding evalu...
Leon Lufkin, Cutter Dawes, Alexander Speicher et al.
· 0 citations