Large Language Models (LLMs) have enabled diverse AI applications. However, LLM inference imposes unprecedented computational and memory overhead, creating an inherent tradeoff between latency Service Level Objectives (SLOs) and resource constraints. Serverless computing, with on-demand provisioning and pay-as-you-go billing, is becoming a promising paradigm for LLM serving. But existing solutions fail to integrate state-of-the-art inference optimizations, resulting in suboptimal GPU utilization and prolonged latency. While Prefill-Decode (PD) disaggregation combined with continuous batching has resolved such inefficiencies in traditional cloud deployments, migrating these techniques to serverless makes two challenges particularly pronounced: (1) SLO-constrained resource provisioning for independently scaling prefill and decode phase functions, and (2) function lifespan management to mitigate resource waste from continuous batching-induced prolonged instance lifespans. To tackle these issues, we propose CELLServe, an SLO-aware and cost-efficient serverless LLM serving system that pioneers integrating PD disaggregation and continuous batching into serverless platforms. CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources. Comprehensive evaluations on five mainstream LLMs and real-world traces show that CELLServe achieves 1.71-1.85×higher request throughput than baselines under identical SLOs and GPU budgets, while sustaining high resource efficiency under dynamic workloads.
Ze-Jian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
Sparse Mixture-of-Experts (MoE) architectures scale LLM capacity, but serving them with expert parallelism often suffers from the straggler effect caused by skewed token routing and uneven placement of hot experts across GPUs. Existing methods mitigate stragglers by adjusting token-to-expert distributions or by replicating hot experts across GPUs, but they either trade output quality for lower tail latency or rely on an expert placement tuned to a specific routing profile. However, real-world LLM serving is multi-task and time-varying, so a placement that works well for one profile can become inefficient as the workload shifts. Motivated by the observation that routing patterns are task-specific, we propose MigMoE, a task-aware expert migration framework that dynamically adjusts expert placement for multi-task expert parallel MoE inference to balance loads across GPUs. In the offline phase, MigMoE profiles each task type and generates a migration plan under a migration budget using our Dynamic Programming-based planning algorithm. In the online phase, MigMoE matches incoming requests to the most suitable task profile and switches to the corresponding expert placement by executing the planned migrations. Experiments on three MoE LLMs show that MigMoE consistently outperforms two state-of-the-art baselines, EPLB and HarMoEny, improving throughput by up to 56.4% and 68.4%, respectively, while alleviating stragglers and maintaining comparable answer quality.
Xu Han, Zinuo Cai, Zhuo-Long Jiang et al.· Proceedings of the Internati...· 0 citations
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
