BYSTANDER: State-Aware End-to-End Latency Prediction for Heterogeneous LLM Inference Scheduling

Abstract

Large Language Model (LLM) inference services increasingly rely on heterogeneous GPU clusters to balance cost and performance. However, request routing in such environments is challenging because schedulers must account for hardware heterogeneity, dynamic workload characteristics, and bursty arrivals. Existing approaches either ignore hardware differences, rely on static heterogeneity-aware allocation, or use queue-based proxies that fail to capture the true remaining work under load. We present BYSTANDER, a prediction-based scheduling framework that uses a small language model (SLM) to estimate End-to-End (E2E) latency of request for each GPU pool from request features and current pool state. BYSTANDER then adaptively narrows the candidate pools via Fisher–Jenks grouping and performs queue-aware selection within that set. Its pool-based design keeps prediction overhead scalable as cluster size grows. We evaluate BYSTANDER on ShareGPT and LMSYS-Chat workloads over heterogeneous clusters of 3–7 GPUs (RTX 3090/4090/5090). Compared with round-robin, oracle-weighted round-robin, shortest-queue-first, and SLM Adaptive baselines, BYSTANDER reduces P99 E2E latency by up to 63.4% and P99 time-to-first-token by up to 87.4% under bursty traffic.