Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining
This work proposes TSA, which interpolates between the raw final iterate and the average of recent checkpoints to balance recent progress against terminal variation and demonstrates that the resulting gains transfer to depth-22 NanoChat, where the combined schedule and estimator improve validation quality.