#artificial intelligence
Sep 2025
Positional Encoding via Token-Aware Phase Attention
Token-Aware Phase Attention (TAPA) is introduced, a new positional encoding method that incorporates a learnable phase function into the attention mechanism and attains substantially lower perplexity and stronger retrieval performance in the long-context regime than RoPE-style baselines.
Yu Wang, Sheng Shen, Rémi Munos et al.
· arXiv.org · 2 citations