Output Embedding Centering for Stable LLM Pretraining
This paper proposes output embedding centering (OEC) as a new mitigation strategy, and demonstrates that it suppresses output logit divergence and finds that $\mu$-loss is significantly less sensitive to regularization hyperparameter tuning than z-loss.