Zhi-Yuan Guo

Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference

LLM inference is constrained by GPU memory capacity and bandwidth. Tiered memory architectures mitigate this by allowing the GPU to offload memory to the remote tier. However, existing memory offloading frameworks rely on prefetching data into local GPU HBM. This approach underutilizes system resources by introducing H...