Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry
Counting is a simple task on which transformers can nevertheless fail at long sequence lengths. We study the cost of counting in a single attention layer, up to a maximum length $L$, focusing on how softmax attention and layer normalization shape the representation from which the count is recovered. We mainly focus on...
A. Shaffy, Manoj Dhakal, Safal Shrestha et al.
· 0 citations