An equality condition for the Dobrushin bound on attention rollout and how often it holds in trained transformers
The Dobrushin coefficient of each attention-rollout factor satisfies $\kappa(\frac12(I+A))\le\frac12(1+\kappa(A))$, and multiplying these inequalities over layers bounds the coefficient of the whole rollout. We characterise exactly when the layerwise bound is tight: equality holds if and only if some token pair attaini...