Jia-Nan Wang

Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

A three-regime batch-size phase diagram that reveals how the role of momentum changes with batch size is revealed, with Polyak enlarging the critical batch size, the largest batch size preserving the best small-batch data-scaling exponent, thereby enabling greater parallelism without sacrificing data efficiency.