UBEP (Unified-Bus Expert Parallelism), a production-ready communication library that rethinks MoE's All-to-All primitives for modern superpod architectures, and reduces All-to-All latency by up to 52% and MoE inference Time Per Output Token (TPOT) by up to 11.1%.
Yi-Peng Liu, Chang Liu, Sitan Shen et al.· Conference on Applications,...· 0 citations
CubeTrace is presented, a unified, function-level flow tracing system that enables microscopic tracing inside heterogeneous cloud gateways and introduces minimal overhead, consuming less than 1% of memory resources and adding less than 1% to forwarding latency.
Yunming Xiao, Yinchao Yang, Jiaqi Zheng et al.· Conference on Applications,...· 1 citation
CoRun is presented, a scheduling-based system that achieves deterministic inference without requiring batch invariance, and employs isolated prefill and fixed-shape batched decode to handle the two stages of LLM inference, respectively, leveraging CUDA graphs for efficient execution and simplified implementation.
Shiju Zhao, Jiacheng Yang, Qi-Hang Chen et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
