Rate-Assured GPU Inference for 5G AI Slices

Efficient LLM Coserving at the Edge via Resource-Aware Cooperative Scheduling

Large language models (LLMs) are increasingly deployed on edge nodes to support edge intelligence applications. To overcome limited GPU memory, offloading-based methods partition model parameters between the GPU and host memory, enabling inference on commodity hardware. However, deploying a single model instance using the offloading-based method often results in significant infrastructure overhead and the underutilization of CPU, GPU, and PCIe resources due to a persistently idle CPU, bursty workload patterns, and bandwidth–compute mismatches. To address this issue, this article proposes RACS, a resource-aware cooperative scheduling (RACS) framework that enables a single edge node to coserve a latency-critical high-priority model and a latency-tolerant low-priority model. The key insight is that PCIe bandwidth constitutes the primary bottleneck in offloading-based inference. RACS comprises a runtime state manager that monitors PCIe availability in real time and a resource-aware cooperative scheduler that orchestrates the low-priority model accordingly. When the high-priority model is active, RACS restricts low-priority execution to preloaded feed-forward layers to avoid PCIe contention. When PCIe is idle, RACS aggressively utilizes GPU and PCIe resources while cooperatively scheduling computations on the CPU to maximize throughput. Extensive experiments with the OPT-13-B and OPT-6.7-B models under diverse prompt lengths, generation lengths, and real-world request traces demonstrate that RACS improves the throughput of offline tasks by up to 27.4% without compromising the latency of the high-priority model.

Learning to schedule: interference-aware optimization for edge AI inference on shared GPUs

A time-varying integer program to minimize the long-term total cost of the edge AI inference system, including the inference latency, the inference error rate, the query-dispatching communication cost, and the energy consumption, subject to resource and workload constraints is proposed.

Breaking the Tradeoff: Elastic and Isolated GPU Sharing with GVM

Weave: Fine-Grained Dynamic SM Scheduling in an MoE Megakernel for Compute-Communication Overlap

Mixture-of-Experts (MoE) inference under expert parallelism (EP) turns each MoE layer into a distributed computation with costly dispatch and combine communication. State-of-the-art systems reduce this cost through communication-computation overlap, splitting the GPU's SMs for communication and computation respectively. However, this approach still leaves GPU resources wasted along two dimensions. Spatially, the best SM split is determined by each layer's routing result and varies across layers and GPUs, so fixed policies mismatch the workload and waste either NVLink bandwidth or compute throughput. Temporally, complex MoE data dependencies introduce bubbles that leave SMs idle. We present Weave, to our knowledge the first MoE overlap system that performs fine-grained dynamic SM scheduling - deciding per layer and per GPU by routing results at runtime. Once routing completes, each layer's communication and computation volumes become known; Weave exploits this predictability through a lightweight cost model running inside the persistent megakernel: a spatial scheduler partitions SMs into communication workers and computation workers to match the communication/computation throughput ratio, and a temporal scheduler coordinates the two worker groups to minimize SM idleness. On 4x H100 SXM GPUs across six mainstream MoE models, Weave achieves a 2.89x geometric-mean MoE-layer speedup and a 1.33x geometric-mean end-to-end speedup over five state-of-the-art baselines.

AsymFlow: Enabling Long-Context LLM Serving via CPU-GPU Prefill-Decode Disaggregation

Driven by cost and privacy constraints, many organizations deploy large language model (LLM) inference on local CPU-GPU servers. For long-context requests, the key-value (KV) cache grows linearly with sequence length and can rapidly exhaust GPU memory, reducing both the maximum supported context and request concurrency. Meanwhile, we observe that GPU utilization is high during the compute-heavy prefill phase but drops sharply during the memory-bound decode phase, where modern CPUs equipped with matrix units can achieve competitive attention throughput. We present AsymFlow, a prefill-decode disaggregated serving system that runs prefill on the GPU and decode on the CPU. AsymFlow (1) streams per-layer KV states through a shared-memory KV pool to overlap KV transfer with GPU prefill, (2) employs a task-aware online dispatcher that jointly accounts for pipeline queueing and KV capacity to prevent GPU OOM and idle bubbles, and (3) accelerates CPU decode attention with AMX. Implemented on SGLang and evaluated on a CPU-GPU platform, AsymFlow serves 32K-context workloads that can trigger OOM on a GPU-only baseline, improves throughput by 1.08–1.25 × over the GPU-only baseline, and by 1.32-2.49 × over a state-of-the-art KV-offloading method; the gains increase as GPU memory becomes tighter.

NUNA: Characterizing and Mitigating Non-Uniform Network Access in Multi-Die GPU Scale-Up Systems

This work highlights the need for NUNA-aware routing (NAR), choosing optimized, spatially-aware inter-GPU paths in large scale-up network topologies and introduces NUNA-aware placement (NAP), placing threadblocks and data near I/O to optimize the inter-GPU traffic.