With the ever-growing size of deep learning models, GPU memory is prone to being insufficient during training. A prominent approach is ZeRO-Offload, which moves the optimizer states to CPU memory and performs parameter update using CPU. However, the deficiencies of ZeRO-Offload include low GPU utilization, imperfect ov...
Zhi-Yi Yao, Zu-Ning Liang, Yue-Dong Xu et al.· 0 citations
This work introduces Schur Replay, a scale-selection algorithm that reproduces the GPTQ updates caused by each block scale and scores the resulting block error after accounting for compensation from unquantized columns.
Rui-Ying Ding, Jie Li, Kang He et al.· 0 citations
DualDecoder is presented, a lightweight serving system for long-context LLM inference that enables efficient sparse KV cache retrieval from host memory that leverages a novel dual-token decoding pipeline that accurately identifies critical KV entries with negligible computational overhead.