Attention Mean Fields Predict Average Representation Dynamics and Reveal Context-Specific Computation
A mean-field analysis of attention, which shows how the average attention from one token to another defines a kernel that carries representations layer to layer and can be iterated through the network to model how the geometry is transformed.
Micah Adler, J. Byers, Mark Crovella
· 0 citations