ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling
This work investigates the expressiveness of general query/key rotations and finds that any normalized continuous positive-definite attention modulation function can be approximated by random rotations induced by its own Fourier transform, which is term Random Fourier Rotations.
Yiwen Chen, J. Ainslie, Krzysztof Choromanski et al.
· 0 citations