Chat-Fusion: An Efficient Batch Inference System for Cloud LLMs Serving
Chat-Fusion is built, an efficient redundancy-free batch inference system that improves the throughput of popular LLMs by 1.4 times compared to other state-of-the-art inference systems, such as ORCA and vLLM.