Cheng-Zhi Lu

ReliefServe: Relieving GPU Pressure in Multi-Model Serving via Selective CPU Escape

Sharing GPUs among many deep learning models is crucial for cost-efficient inference, but bursty multi-model workloads can easily overwhelm GPU capacity, causing severe tail latency and SLO goodput drops. Existing solutions—whether traffic-aware scheduling or hardware-level resource partitioning—can only juggle content...