Aravind Ramana Venkatramanan

The Geometry of Language Model Logits: Generic Structure and Learned Departures

This work empirically identifies that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation across prediction contexts, architectures, and scales, emerging progressively during pretraining.