Jun 2026
Structure Before Collapse: Transient semantic geometry in next-token prediction
This work finds that semantic geometry emerges early in training, and that representations cluster by shared attributes despite receiving no explicit supervision to do so, and proposes a preliminary modification to the commonly used unconstrained features model to capture the emergent semantic geometry.
Yize Zhao, Isabel Papadimitriou, Christos Thrampoulidis
· arXiv.org · 1 citation