Chat-Fusion: An Efficient Batch Inference System for Cloud LLMs Serving

Abstract

Batch processing is useful for optimizing the utilization of various hardware resources, such as GPUs, CPUs, and other specialized accelerators, and is also important for improving the throughput of LLM serving. Many research works have been conducted for better batching and scheduling policies for the efficiency of cloud LLM serving systems, enhancing data locality and resource utility. In our analysis of actual cloud LLM traces, we observed substantial computational and memory inefficiencies when handling batches of simultaneous user prompts. This inefficiency arises from padding and representation-equivalent tokens across prompts, which can lead to repeated or unnecessary operations. Existing LLM inference systems ignored such optimization opportunities offered by redundant tokens in the batch, resulting in a large amount of redundant computation and storage, especially the waste of KV cache memory, which severely limits the batch size and fails to fully utilize the potential of parallel computing in GPUs. To address these problems, we build Chat-Fusion, an efficient redundancy-free batch inference system. At the scheduling level, Chat-Fusion includes (1) a dynamic batching strategy that avoids padding and misalignment due to varied prompt length, (2) a runtime for clustering tokens into a repetition-free dense format and an indexing mechanism for prompt restoration, and (3) a compact KV-cache layout strategy that avoids fragmentation. At the execution level, since the inference process is changed by compression, we reimplemented the necessary transformer and KV-cache operations to realize high-efficiency, redundancy-free LLM computation on condensed input requests. Our evaluation shows that at the same latency, Chat-Fusion improves the throughput of popular LLMs by <inline-formula><tex-math notation="LaTeX">$1.1-3.4\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>1</mml:mn><mml:mo>.</mml:mo><mml:mn>1</mml:mn><mml:mo>-</mml:mo><mml:mn>3</mml:mn><mml:mo>.</mml:mo><mml:mn>4</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="yu-ieq1-3734339.gif"/></alternatives></inline-formula> compared to other state-of-the-art inference systems, such as ORCA and vLLM.