Kai Zhang

Chat-Fusion: An Efficient Batch Inference System for Cloud LLMs Serving

Batch processing is useful for optimizing the utilization of various hardware resources, such as GPUs, CPUs, and other specialized accelerators, and is also important for improving the throughput of LLM serving. Many research works have been conducted for better batching and scheduling policies for the efficiency of cl...