Rethinking LLM Deployment for Intent-Based Serving
TL;DR
MaverIQ is an intent-based LLM inference serving system that automatically maps user intents to deployment configurations while minimizing operational cost for the provider and reduces profiling overheads by 7-15× compared to state-of-the-art baselines.
Abstract
Large Language Models (LLMs) expose a large deployment configuration space spanning parallelism and compression techniques, with each configuration introducing different tradeoffs in latency, memory consumption, cost, and output quality. Existing systems either rely on expensive profiling across deployment configurations or inefficiently utilize fragmented GPU resources in multi-tenant clusters. We present MaverIQ, an intent-based LLM inference serving system that automatically maps user intents to deployment configurations while minimizing operational cost for the provider. To reduce profiling overheads, MaverIQ introduces lightweight LLM fingerprints and analytical models that extrapolate latency and memory footprint from only a few observations. To efficiently utilize fragmented GPU resources, MaverIQ leverages our observation that, unlike training, unevenly distributing LLM layers across GPUs has little impact on inference latency. Our evaluation shows that MaverIQ reduces profiling cost by 7-15× compared to state-of-the-art baselines and reduces operational cost by 3.8-8.3× across diverse LLMs, traces, and loads while effectively meeting user intents. Our code is available at https://github.com/UT-SysML/MaverIQ.