CoScale is proposed, a layer-wise dynamic scaling mechanism that selectively expands the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism without altering model architectures or adding hardware overhead.
Abstract
Online large language model (LLM) serving has become the backbone of modern AI applications, powering diverse downstream services through shared hardware clusters. However, modern serving systems frequently encounter highly dynamic workloads characterized by severe workload skewness, where a small fraction of model instances receives the vast majority of traffic. Existing instance-level scaling mechanisms are limited by coarse-grained resource adjustment: scaling up requires the cold-start of full-model replicas, incurring substantial latency, while scaling down leaves the system vulnerable to performance degradation during sudden traffic surges. The key insight of this work is that LLM serving offers a unique opportunity for fine-grained scaling. In this paper, we propose CoCoScale, a layer-wise dynamic scaling mechanism that selectively expands the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism without altering model architectures or adding hardware overhead. Evaluations demonstrate that CoCoScale significantly reduces cold start latency by 97.9%-99.3% compared to traditional scale up. Under production traces, CoCoScale reduces average latency by 20.7\%--28.1\% and achieves full Service Level Objective (SLO) attainment, demonstrating superior dynamic adaptability and resource efficiency.
OpScale is presented, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving that attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.
Xingqi Cui, Chieh-Jan Mike Liang, Ziang Tang et al.· 0 citations
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
Existing Large Language Model (LLM) inference systems often rely on static model placement and scheduling policies, which struggle to handle heterogeneous and dynamic real-world workloads. The key challenge is to adapt serving strategies to workload fluctuations while keeping reconfiguration overhead minimal. In this paper, we present OrionInfer, an adaptive LLM serving system that aligns inference strategies with real-time demand. OrionInfer introduces three key techniques: (1) runtime switching between data parallelism and tensor parallelism with negligible overhead; (2) an efficient inference pipeline that preserves batching efficiency during parallelism transitions; and (3) live-migration-based load balancing to alleviate memory pressure and improve resource utilization. Evaluations across multiple model scales show that OrionInfer delivers robust performance under diverse serving scenarios. In end-to-end serving, it reduces average TTFT by up to 25% over DP-priority configurations under low loads and lowers P99 tail latency by 50%--90% over TP-priority configurations under most high-traffic settings. In disaggregated prefill serving, OrionInfer improves prefill completion time (PCT) SLO attainment by up to 16.5 percentage points over DP-priority static baselines and reduces P99 PCT by up to 74.7% over TP-priority static baselines. Compared with dynamic baseline, OrionInfer provides better tail-latency stability, reducing P99 PCT by 38.6%--40.8% while avoiding the extra memory footprint.
Jingqi Feng, Guang Yang, Yukai Huang et al.· Proceedings of the 32nd ACM...· 0 citations
The rapid adoption of large language models (LLMs) has increased the need for efficient multi-tenant inference systems that maximize GPU utilization. However, existing frameworks struggle to scale due to the high memory demands of model weights and key-value (KV) caches. We present DynamoServe, a multi-tenant LLM serving framework that addresses these challenges through three key innovations: (1) leveraging stranded GPU memory to offload model weights and KV caches, (2) mitigating resource fragmentation in multi-workload environments, and (3) improving memory locality through coordinated data placement and demand-driven weight migration across GPUs. Together, these techniques enable high-throughput, low-latency inference. Experiments on state-of-the-art models show that DynamoServe significantly improves memory efficiency without sacrificing latency.
Diman Zad Tootaghaj, Khaled Diab, Bob Lantz et al.· 0 citations
Large language model (LLM) serving commonly increases batch size to improve throughput, but performance eventually reaches a deployment-dependent plateau beyond which larger batches provide marginal gains while increasing latency and GPU memory consumption. Previous studies have attributed this behavior to HBM/DRAM bandwidth limitations, but the underlying causes have primarily been supported by conceptual arguments or high-level performance observations. As our first contribution, we present a detailed GPU characterization using hardware profiling techniques, demonstrating that throughput saturation originates in the attention kernels during the decode phase. Specifically, we show that their nearly constant arithmetic intensity as active-context lengths increases -not merely larger batch sizes- drives DRAM-bandwidth saturation, while the achieved compute throughput remains far below the hardware limit. Building on this analysis, we present the Batching Configuration Advisor (BCA), which selects the highest-throughput batching configuration satisfying a target latency constraint and identifies up to 55 GB of GPU memory allocation that can be avoided for the evaluated OPT models with minimal throughput loss. To enable these recommendations, we introduce SLIM (Saturation-Aware Lightweight Performance Model), a semi-analytical model that predicts LLM inference throughput and latency from analytical formulations of Transformer computation and memory traffic. Across the evaluated scenarios, SLIM outperforms representative performance-modeling baselines while successfully generalizing to previously unseen operating conditions.
Pol G.Recasens, F. Agulló, Yue Zhu et al.· 0 citations
HYDRA jointly explores chiplet composition, placement, inter-chiplet bandwidth provisioning, dynamic batching, dynamic batching, and runtime scheduling, and a fast Markov-based performance estimator that captures multi-tenant runtime dynamics for efficient and accurate exploration.
Jiahao Lin, Alish Kanani, Sang-Won Lee et al.· 0 citations