Jia-Hui Ni

Queuing for Inference: Dynamic Memory Footprints and Costly Preemption

A new queuing model that captures the impacts of transformer-based neural networks features on system stability and scheduling design, and proposes the Multi-Level-Priority-Threshold (MLPT) policy, that limits preemption while preserving priority for short jobs.