Distance generalization in transformers: why bother with positional encoding?
A thorough investigation of distance generalization probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length, finding that it is paramount to improve the understanding of the underlying mechanisms.
D. Nevermann, Claudius Gros
· 0 citations