Aug 2026· Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication· pp. 1075-1094· 0 citations· 87 references
Computer Science
TL;DR
LEVELLER is proposed, the first communication scheduling system that achieves max-min fairness specifically for DLT workloads and introduces a novel online metric, normalized progress rate, which quantifies training experience by measuring actual progress against a contention-free ideal.
Abstract
The rapid expansion of Large Language Models (LLMs) has established Deep Learning Training (DLT) as a major workload in modern multi-tenant GPU clusters. While fair computational scheduling is well-studied, fair communication scheduling in handling execution-time network contention remains underexplored. Current DLT communication schedulers focus primarily on efficiency-oriented objectives, often causing disparate performance degradation or even job starvation when concurrent jobs compete for shared bandwidth. In this paper, we propose LEVELLER, the first communication scheduling system that achieves max-min fairness specifically for DLT workloads. To bridge the gap between traditional flow-level fairness and DLT job-level fairness, LEVELLER introduces a novel online metric, normalized progress rate, which quantifies training experience by measuring actual progress against a contention-free ideal. We theoretically prove that a greedy, progress-rate-aware flow scheduling strategy achieves at least a 1/2-approximation of max-min fairness for arbitrary workloads, even with lazy priority updates. Building on theoretical results, LEVELLER employs preemptive, dynamic-priority scheduling to achieve near-optimal fairness on commodity infrastructure without hardware or protocol modifications. We prototype LEVELLER on both RDMA and TCP, evaluating it through a physical GPU cluster and trace-based simulations. Experimental results on 10 LLMs show that LEVELLER improves the minimum progress rate by 57% and fairness by 17% compared to state-of-the-art solutions (including CASSINI, CRUX, Symphony and Sincronia), while maintaining competitive cluster efficiency.
Multi-tenant GPU clusters frequently remain underutilized even when tenants experience long queueing delays, because quota control, queue ordering, preemption, and GPU sharing are driven by different local signals. We present DeepShare, a scheduler that uses a continuous tenant-assurance signal to coordinate these decisions at runtime. DeepShare combines elastic quota borrowing, tenant-specific runtime prediction, cost-aware best-effort preemption, and interference-aware MPS colocation, while using the same assurance signal to decide when borrowed capacity should be reclaimed and when sharing should become more conservative. In trace-driven experiments on 23,859 Venus jobs and 3,200 internal jobs, DeepShare achieves an average GPU utilization of 70.58%, a 29.5% improvement over the strongest non-intrusive sharing baseline, while reducing average queueing delay by 46%. On a 16-GPU Kubernetes testbed, it reduces the average job completion time by 34% and maintains 93% QoS compliance for guaranteed tenants. These results show that treating tenant assurance as a runtime control loop achieves a more advantageous utilization-QoS trade-off than optimizing quotas, scheduling, and resource sharing independently.
Jing-Hao Wang, Yi-Hang Zhou, Xiao Zhou et al.· 0 citations
LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and batching fairness. However, these approaches do not provide latency isolation guarantees; as a result, well-behaved clients can still experience significant degradation to their token-level latencies. In this paper, we present FairInference, which provides the novel {\delta}-token fairness guarantee: for a well-behaved client, if a token is generated in d time units in isolation, it will be generated within d + {\delta} time units in multi-tenant execution, providing strong latency isolation guarantees for LLM serving. To achieve this, FairInference addresses a key challenge of LLM serving: bounding delays from sharing GPU resources without support for fine-grained scheduling or resource allocation. In FairInference, the scheduler enforces per-token deadlines, while bounding the delays from GPU compute sharing and accounting for the additional delays introduced by the shared KV caching in GPU memory. We show that FairInference effectively bounds token-level latency spikes for well-behaved clients and improves overall throughput compared to state-of-the-art LLM serving systems.
Dev Bali, Soujanya Ponnapalli, Yi-Chu Wang et al.· 0 citations
A mathematical scheduling model that connects within-batch resource fairness to system throughput and provides a bi-criterion scheduling policy, ISJL, which maintains high throughput while aligning max-driven batch cost with token-metered revenue.
Large language models (LLMs) are increasingly deployed on edge nodes to support edge intelligence applications. To overcome limited GPU memory, offloading-based methods partition model parameters between the GPU and host memory, enabling inference on commodity hardware. However, deploying a single model instance using the offloading-based method often results in significant infrastructure overhead and the underutilization of CPU, GPU, and PCIe resources due to a persistently idle CPU, bursty workload patterns, and bandwidth–compute mismatches. To address this issue, this article proposes RACS, a resource-aware cooperative scheduling (RACS) framework that enables a single edge node to coserve a latency-critical high-priority model and a latency-tolerant low-priority model. The key insight is that PCIe bandwidth constitutes the primary bottleneck in offloading-based inference. RACS comprises a runtime state manager that monitors PCIe availability in real time and a resource-aware cooperative scheduler that orchestrates the low-priority model accordingly. When the high-priority model is active, RACS restricts low-priority execution to preloaded feed-forward layers to avoid PCIe contention. When PCIe is idle, RACS aggressively utilizes GPU and PCIe resources while cooperatively scheduling computations on the CPU to maximize throughput. Extensive experiments with the OPT-13-B and OPT-6.7-B models under diverse prompt lengths, generation lengths, and real-world request traces demonstrate that RACS improves the throughput of offline tasks by up to 27.4% without compromising the latency of the high-priority model.
Zhen-Zheng Li, Zhiqing Tang, Jian-Xiong Guo et al.· IEEE Internet of Things Jour...· 0 citations
Interactive services typically over-provision CPU resources to meet Service Level Objectives (SLOs) for tail latency amidst workload fluctuations. This inefficiency motivates emerging research into workload co-location, where batch jobs are hosted alongside interactive services to harvest underutilized resources. However, the Linux Completely Fair Scheduler (CFS) limits potential resource efficiency gains. CFS’s fairness-oriented design lacks support for workload-specific scheduling policies and is unable to simultaneously enforce performance isolation while facilitating fine-grained resource sharing across co-located workloads. Therefore, we present G1Stack, a scheduler framework designed for workload co-location. Specifically, G1Stack incorporates: (1) A parallel dual-policy architecture that schedules latency-critical (LC) tasks from interactive services and best-effort (BE) tasks from batch jobs separately, guaranteeing responsiveness for the former while enhancing computational throughput for the latter; (2) A learning-assisted load-balancing approach that dynamically interleaves LC and BE tasks across cores with fine-grained temporal and spatial distribution to optimize resource efficiency; (3) Integrated workload-aware auto-scaling and load-shedding mechanisms to minimize latency during up-scaling under load spikes, ensuring system responsiveness under dynamic workloads. Stress-testing results demonstrate that G1Stack reduces the non-productive resource ratio to at most 7.55% and shortens the average completion time of co-located batch jobs by up to 62.45% compared to baselines.
Di-Shi Xu, Fagui Liu, Bin Wang et al.· IEEE transactions on compute...· 0 citations
Concord, a novel GPU sharing-enabled workload scheduler that outperforms state-of-the-art schedulers, achieves a 1.68 × reduction in JCT and a 29% improvement in GPU utilization in high-load scenarios.
Xinhua Wang, Wei-Wei Lin, Hai-Jie Wu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
