How Local Mixing Encodes Relative Position in Global NoPE Attention
An explanation of how hybrid models of this sort can implicitly encode position at global NoPE layers is developed, and insights are provided for how to encode position in a way that can extrapolate to longer sequence lengths indefinitely.
Cutter Dawes, Nick Alonso, Tomas Figliolia et al.
· 0 citations