Xiao-Hai Hu

Decomposing Predictive Kubernetes Autoscaling for Large Language Model Serving Under Long Startup Delays

Large language model (LLM) inference deployed on Kubernetes faces an autoscaling challenge that conventional web services do not: new serving replicas take two to ten minutes to start because multi-gigabyte model weights must be loaded, which makes purely reactive scaling structurally late. We ask a sharp question: amo...