Jul 2026· ACM SIGOPS Operating Systems Review· Vol 60, pp. 30 - 40· 0 citations· 47 references
Computer Science
TL;DR
This paper proposes a deployment scheme for agentic workloads tailored for serverless, accompanied by pre-warming policies that minimize the idle resource footprint and startup latencies and outlines promising research directions for serverless agents.
Abstract
Accelerating generative AI adoption has driven the expansion of data centers, which amass GPUs, DRAM, and SSDs to feed emerging, resource-hungry AI workloads. The serverless cloud model offers a path to improve application resource efficiency by loading instances on demand. However, the suitability of emerging AI workloads for serverless remains insufficiently explored. We survey the state-of-the-art in serverless hosting for LLM applications and find that: (1) Despite advances in serverless LLM hosting, model loading and initialization processes still dominate startup latency. (2) Agentic AI workloads have not yet been characterized under the serverless context. We propose a deployment scheme for agentic workloads tailored for serverless, accompanied by pre-warming policies that minimize the idle resource footprint and startup latencies. This paper outlines promising research directions for serverless agents.
This work organizes agentic workflows in a taxonomy and presents its first architectural characterization with a production study at Microsoft Azure and a controlled study of open-source frameworks, showing that agentic execution is fragmented and heterogeneous.
Ji-Rong Yang, Peizhe Liu, Chaojie Zhang et al.· 2 citations
Large Language Models (LLMs) have enabled diverse AI applications. However, LLM inference imposes unprecedented computational and memory overhead, creating an inherent tradeoff between latency Service Level Objectives (SLOs) and resource constraints. Serverless computing, with on-demand provisioning and pay-as-you-go billing, is becoming a promising paradigm for LLM serving. But existing solutions fail to integrate state-of-the-art inference optimizations, resulting in suboptimal GPU utilization and prolonged latency. While Prefill-Decode (PD) disaggregation combined with continuous batching has resolved such inefficiencies in traditional cloud deployments, migrating these techniques to serverless makes two challenges particularly pronounced: (1) SLO-constrained resource provisioning for independently scaling prefill and decode phase functions, and (2) function lifespan management to mitigate resource waste from continuous batching-induced prolonged instance lifespans. To tackle these issues, we propose CELLServe, an SLO-aware and cost-efficient serverless LLM serving system that pioneers integrating PD disaggregation and continuous batching into serverless platforms. CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources. Comprehensive evaluations on five mainstream LLMs and real-world traces show that CELLServe achieves 1.71-1.85×higher request throughput than baselines under identical SLOs and GPU budgets, while sustaining high resource efficiency under dynamic workloads.
Ze-Jian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
This work introduces WASP, a configurable framework that brings stateful serverless execution to the edge-cloud continuum by abandoning monolithic architectures in favor of strictly decoupled, pluggable components, and lets system administrators swap the WASM runtime and the datastore to fit available resources and application requirements, without altering application code.
Text-to-image (T2I) workflows are increasingly deployed on serverless platforms because users often compose customized workflows and invoke them intermittently. Existing platforms typically deploy each workflow as an opaque GPU function, provisioning, placing, and scaling all constituent models in the workflow together. This monolithic design obscures workflow structure, inflates scaling overhead, forces users to manage low-level GPU coordination, and limits fine-grained fairness in multi-tenant clusters. In this paper, we present ServerlessT2I, a serverless-native system that decomposes a T2I workflow into loosely coupled model functions that can be independently managed and scheduled. By explicitly managing individual model execution, ServerlessT2I enables per-model scaling, declarative workflow composition, transparent GPU-resident communication, and fairness-aware scheduling. To make this decomposition efficient, ServerlessT2I harvests slack GPU memory left idle by compute-bound T2I inference to build a data plane that reduces model loading and data communication overheads. \sys{} further introduces a fair scheduler for multi-tenant serving. Using production traces, ServerlessT2I sustains up to 2$\times$ higher request rates than existing T2I workflow serving systems with the same GPU budget; for a fixed request rate, it saves up to 3$\times$ GPU resources while satisfying service level objectives (SLOs).
Xiaoxiao Jiang, Suyi Li, Sheng Yao et al.· arXiv.org· 0 citations
Public LLM services serve diverse multi-agent applications with varying workflow dependencies and performance requirements. Requests generated by these applications often exhibit commonality and interdependence, yet current systems largely ignore such application-level structure. As a result, at the LLM engine cluster level, assigning requests to engines with the shortest queue can cause inefficient KV-cache transfers across GPUs. Using three representative multi-agent applications, we show that current scheduling methods miss opportunities to (a) improve performance through KV-cache reuse and reduced data transfer, and (b) increase goodput via batch management informed by workflow dependencies. Motivated by these observations, we propose a Workflow-Aware Scheduling system for Multi-Agent LLM systems (WaSMa) that incorporates cluster-and engine-level scheduling to optimize LLM request execution across GPU resources. Experimental results show that WaSMa reduces the P95 tail latency by up to 48% and improves goodput by up to 107% compared to existing methods.
Uttam Rao, Ali Zafar Sadiq, Hai-Ying Shen et al.· International Conference on...· 0 citations
Agentic LLM workflows issue many dependent calls with unpredictable resource demand, causing queue buildup and latency degradation on shared serving backends when left unmanaged. In this paper, we propose CALM-MAS, a congestion-aware serving framework for LLM applications that treats LLM test-time computation as an elastic resource, dynamically adjusting the compute profile of admitted tasks to tame congestion. CALM-MAS detects early signals of back-end saturation, and leverages the flexibility of LLM applications to regulate load. During spikes of requests, the system downgrades agent topology and reasoning depth; during low-utilization periods, it allocates additional reasoning effort to maximize task accuracy. Compared with a static serving baseline based on vLLM, CALM-MAS reduces shared-backend tail latency by 77% with the accuracy degradation remaining confined to 6.1 pps relative to the native agent configuration.
Mouheb Ben Nasr, Muhammad Bilal, Alessandro Cornacchia et al.· Proceedings of the 17th ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
