Large Language Models (LLMs) expose a large deployment configuration space spanning parallelism and compression techniques, with each configuration introducing different tradeoffs in latency, memory consumption, cost, and output quality. Existing systems either rely on expensive profiling across deployment configurations or inefficiently utilize fragmented GPU resources in multi-tenant clusters. We present MaverIQ, an intent-based LLM inference serving system that automatically maps user intents to deployment configurations while minimizing operational cost for the provider. To reduce profiling overheads, MaverIQ introduces lightweight LLM fingerprints and analytical models that extrapolate latency and memory footprint from only a few observations. To efficiently utilize fragmented GPU resources, MaverIQ leverages our observation that, unlike training, unevenly distributing LLM layers across GPUs has little impact on inference latency. Our evaluation shows that MaverIQ reduces profiling cost by 7-15× compared to state-of-the-art baselines and reduces operational cost by 3.8-8.3× across diverse LLMs, traces, and loads while effectively meeting user intents. Our code is available at https://github.com/UT-SysML/MaverIQ.
Dimitrios Liakopoulos, Prasoon Sinha, Tianrui Hu et al.· ACM SIGOPS Operating Systems...· 0 citations
KV cache accelerates LLM inference by avoiding redundant computation, but its rapidly growing memory footprint makes GPU memory a primary bottleneck in modern serving systems. Recent approaches extend GPU memory using CPU memory through KV-cache swapping. However, because KV cache is continuously updated during decoding, swapping introduces substantial synchronization and bidirectional transfer overheads. We present Oneiros, a dynamic remapping engine for multi-tenant LLM serving. Oneiros is based on a simple observation: unlike KV cache, model parameters remain immutable during inference. Instead of swapping KV cache itself, Oneiros dynamically repurposes GPU memory allocated for model parameters as KV cache capacity, enabling nonblocking, unidirectional parameter transfer. This approach is particularly effective in multi-tenant environments, where memory allocated to inactive models can be reclaimed dynamically for active workloads. We implement Oneiros in vLLM and evaluate it on modern GH200 systems. Compared to vLLM, Oneiros reduces tail latency by up to 99.3% and improves throughput by up to 86.7%. Compared to KV-cache swapping approaches, Oneiros achieves substantially higher throughput by avoiding synchronization overheads during runtime memory extension. Source code of Oneiros is available at https://github.com/UT-SysML/Oneiros/1.