Optimizers for Diffusion Models: A Controlled Benchmark
A controlled optimizer benchmark across four diffusion formulations, to the authors' knowledge the first for discrete diffusion, and methods validated on autoregressive language model pretraining transfer well: Muon, MARS-M and SOAP each beat the tuned AdamW on at least one diffusion formulation.
Arman Bolatov, Egor Shulgin, David Li et al.
· 0 citations