Book
Open access
Sep 2026
Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC
Serving offline large language model (LLM) inference workloads (e.g., log summarization and bulk translation) can consume up to 30% of GPUs in production. Despite this significant share, the characteristics of offline inference remain largely understudied. In this paper, we start by analyzing 1.5 million tasks comprisi...
Le-Ping Yang, Xue Li, Kun Qian et al.
· Proceedings of the ACM SIGOP... · 0 citations