LMTracer is presented, a fine-grained and real-time performance profiling framework for production LLM services that embed profiling logic into the execution through graph-embedded probing and streaming buffered profiling data to CPUs on demand to keep the execution of user kernels uninterrupted.
Wei Liu, Yong-Chao He, Bo-Han Zhao et al.· Proceedings of the ACM SIGOP...· 0 citations
FluxMoE is presented, which decouples experts from physical GPU residency and adapts their footprint to available memory through a new \emph{expert paging} abstraction, and streams weights on demand while keeping expert computation on GPUs.
Qing-Xiu Liu, Yong-Chao He, Run-Han Jiang et al.· 0 citations
This work proposes the Communicating Finite Automata (CFA) abstraction to systematically analyze cross-component optimization opportunities, and designs a programming framework to enable CFA-based component program refactoring that preserves the original sequential program structure while enabling safe concurrent compo...