FPDT democratizes long sequence training, making it accessible on mobile and commodity hardware, such as high-end local machines, and trains contexts up to 16x longer than prior sequence-parallel baselines on the same hardware.
Jing-Han Yao, Sam Ade Jacobs, Masahiro Tanaka et al.· GetMobile Mobile Computing a...· 0 citations
ZenFlow is presented, a new offloaded training framework that decouples updates between GPU and CPU by prioritizing important gradients and achieves up to 5× end-to-end speedup, 2× lower PCIe traffic, 85% less GPU stalls, and up to 72% lower cloud GPU cost, all while preserving accuracy.
Ting-Feng Lan, Yu-Sen Wu, Bin Ma et al.· Proceedings of the ACM on Ma...· 8 citations· ⚡3
Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs si...
Ting-Feng Lan, Yu-Sen Wu, Bin Ma et al.· Proceedings of the ACM on Ma...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.