Pick and Spin: Cold-Start-Aware Routing for Self-Hosted LLM Serving

Abstract

Self-hosting multiple large language models on private or hybrid cloud infrastructure is increasingly appealing for organizations seeking privacy, cost control, and customization, yet no existing system coordinates model selection with model lifecycle management. Serving frameworks such as KServe and Ray Serve deploy models but lack query-aware routing. Routing frameworks such as RouteLLM and FrugalGPT select models but ignore whether those models are warm or cold. We present Pick and Spin, a multi-model orchestration framework built on Kubernetes that unifies these concerns. Pick classifies query complexity using a hybrid keyword-DistilBERT approach and selects models via Thompson Sampling, incorporating lifecycle state from Spin. Spin tracks per-model warm and cold states through a Cold Start State Machine, factors I/O-bound weight-loading penalties into routing decisions, and manages warm pools for frequently accessed models. The two components form a closed feedback loop where routing decisions account for resource availability and orchestration anticipates routing patterns. Deployed as a Helm umbrella chart serving nine LLMs (1B–27B parameters) via vLLM on NVIDIA A100 GPUs, Pick and Spin achieves 61% GPU utilization versus 31% for static deployment, reduces GPU-hours by 46%, and maintains a 9.2% per-query cold start rate across 310,190 total inference runs-31,019 queries evaluated against all nine models for ground truth, then routed through Pick and Spin-spanning eight standard benchmarks.