Jul 2026
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
This work modify RoPE by learning a scalar per frequency, treating frequencies as learnable parameters rather than hyperparameters, and observes and analyzes the emergence of a high-norm, positional LeRoPE band.
Petros Karypis, Sean O'Brien, Shreyas Kadekodi et al.
· arXiv.org · 0 citations