Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets
A simple theory-guided method for choosing useful reset periods is derived, showing that in low precision the key question is not only whether resets help, but when they should be applied.