Accelerating Sharded Data Parallelism at Scale with Federated Learning
Inspired by the efficient communication principles of federated learning (FL), this work introduces two hybrid algorithms - FL+FSDP and FL+HSDP - interleaving sharded DP with FedAvg-style aggregations, demonstrating superior computational efficiency and improved model quality.
Gianluca Mittone, Marco Aldinucci
· European Conference on Paral... · 0 citations