Minjae Jung

BYSTANDER: State-Aware End-to-End Latency Prediction for Heterogeneous LLM Inference Scheduling

Large Language Model (LLM) inference services increasingly rely on heterogeneous GPU clusters to balance cost and performance. However, request routing in such environments is challenging because schedulers must account for hardware heterogeneity, dynamic workload characteristics, and bursty arrivals. Existing approach...