Disentangling Representation Evolution in Transformers through Directional Decomposition
It is demonstrated that full-aggregate parallel suppression during from-scratch pretraining lowers validation-loss trajectories and improves downstream averages, with the value-space variant strongest.
Shwai He, Haichao Zhang, Shen Yan
· 0 citations