Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization
A dynamic explanation of how data statistics and architecture jointly shape token embeddings in language models is provided, and an implicit bias in the space of data statistics is revealed: training proceeds from simpler, low-order statistical relations toward increasingly complex, context-dependent ones.
Jun-Jie Yao, Liangkai Hang, Zhi-qin John Xu
· 0 citations