UvA-DARE (Digital Academic

UvA-DARE (Digital Academic

MoSE: Mode-Switching Expander for Mixed LLM Training and Inference

AI clusters increasingly run large language model (LLM) inference and training on the same fabric. Prefill-decode (P-D) disaggregation creates key-value (KV) cache transfers between prefill and decode groups, whereas training collectives and all-to-all traffic benefit from near-uniform global connectivity. A static spa...

Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode

LLM inference decode executes fine-grained collectives whose latency directly limits per-user interactivity, yet scale-up fabrics are still built from electronic packet switches designed for training. We observe that, for a given model deployment and parallelization, the communication structure of decode is fixed, and...

of the 24th USENIX Conference on File and Storage Technologies

SolidAttention is introduced, an LLM inference engine which addresses limitations through a tight co-design of dynamic attention sparsity algorithms and SSD-based storage management and minimizes SSD-induced blocking latency.

LMTracer: Fine-Grained and Real-Time Performance Profiling for Production LLM Systems

LMTracer is presented, a fine-grained and real-time performance profiling framework for production LLM services that embed profiling logic into the execution through graph-embedded probing and streaming buffered profiling data to CPUs on demand to keep the execution of user kernels uninterrupted.

Towards Training Private LLMs: Exploring Fine-Tuning Language Models on Apple Silicon with RDMA over Thunderbolt

This paper characterize RDMA-over-TB communication on Mac Studio nodes, showing that the measured bandwidth is far below nominal TB specifications, and compares Apple Silicon with an NVIDIA H100 platform to quantify the trade-off between memory capacity, throughput, and acquisition cost.