How Inner Optimizers Shape Replica Scaling
It is theoretically and empirically demonstrated that whitening-based pre-conditioning is a powerful inner-optimizer desideratum, and empirical evaluation extends the results to reveal that whitening-based inner-optimizers achieve strong replica scaling (for 8-96 replicas), with performance relative to DP improving as...
Eugene Belilovsky, Stefan Horoi, Edouard Oyallon et al.
· 0 citations