AnchorDL: Dual-Locality-based Scheduling for Serverless Inference Workflows

Abstract

Serverless computing has emerged as an attractive deployment model for deep learning model inference workflows, enabling elastic scaling and fine-grained resource billing across function instances. However, scheduling in this setting introduces a competing-objective challenge: placement decisions simultaneously govern data transfer overhead, determined by whether dependent function instances are co-located, and model loading overhead, determined by whether required model weights are memory-resident on the target node. We present AnchorDL, a joint-cost look-ahead scheduler that minimizes the combined cost of both overheads at each placement decision, with a forward term that avoids greedy suboptimality across adjacent data dependencies. Evaluated against three baselines across chain, fan-in, and fan-out workflow topologies under trace-based workloads, AnchorDL achieves the lowest median end-to-end latency across all evaluated workflows and reduces P90 latency by up to 57.4% against the model-centric baseline in the fan-out workflow. The look-ahead term further contributes substantially beyond greedy joint-cost placement.