Predictive Geometry of Hidden Trajectories in Transformers
These results support a predictive-geometric view of transformer computation: near successful trajectories, the terminal loss induces a thin, anisotropic set of output-relevant hidden-state directions that can be measured and exploited for compression and distillation.
Timur Mudarisov, M. Burtsev, Tatiana Petrova et al.
· 0 citations