LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
This work modify RoPE by learning a scalar per frequency, treating frequencies as learnable parameters rather than hyperparameters, and observes and analyzes the emergence of a high-norm, positional LeRoPE band.