Fast and Memory Efficient Offload Training Framework with Hybrid XPU Computation
This paper designs and implements MemFerry, a novel hybrid on-GPU and DHA model that offloads gradients to CPU memory via DHA, and extends it to emerging scale-up domains with ScaleUp-MemFerry, which exploits otherwise underutilized accelerator interconnect bandwidth to assist host-to-accelerator data movement through...