Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
It is proved that the token distribution rapidly concentrates onto the push-forward of the initial distribution under a projection map induced by the key, query, and value matrices, and remains metastable for moderate times.