Preprint
Jul 2026
Index SLM Technical Report
Control studies on model depth, learning-rate magnitude and scheduling, the interaction between learning-rate decay and data quality, and the effect of including instruction data during pre-training are reported, and an unexplained surge in benchmark performance midway through the constant-learning-rate phase is documented.
Lusheng Zhang, Shien He, Tianxing Yan et al.
· 0 citations