Mixture-of-Experts (MoE) models have become prevailing, yet their distributed deployment suffers from severe dynamic load imbalance due to the conflict between static expert placement and inherent routing dynamism. While host DRAM offers capacity to avoid static device storage for load balancing, we identify existing o...
Wen-Xun Wang, Xiu-Hong Li, Yi-Da Wang et al.· 0 citations
Training LLM requires hybrid parallelism across thousands of GPUs, yet tensor parallelism (TP)—partitioning large weight tensors—remains limited to small degrees in practice due to hardware and software scalability constraints. This limited TP degree forces deeper pipeline parallelism or larger data parallelism, each i...
Zong-Le Huang, Shu-Pei Fan, Xin-Yuan Lin et al.· Proceedings of the 32nd ACM...· 0 citations
This paper introduces RapidMoE, a residual offloading system for efficient large-scale MoE inference, and proposes how RapidMoE leverages a residual-split framework to enable offloading paradigm shift from expert-level to bit-level, which unfolds across three key dimensions.
Wen-Xun Wang, Li-Kai Ma, Zong-Le Huang et al.· 1 citation
End-to-end case studies show that sparse workloads favor criticality-aware heterogeneous pods, whereas LLM inference favors scale-up islands; the resulting designs deliver 6.20 and 2.12 geomean speedups, respectively, while reducing cost and power relative to the baselines.
Yu-Chen Fan, Minghong Sun, Ji-Kui Ma et al.· arXiv.org· 0 citations
H3-Attn is proposed, an Attention-efficient 3D DRAM PNM processor for low-batch LLM inference that features a hybrid head parallelism for Attention processing, whereby various optimized Attention mechanisms with spatial tiled FlashAttention can be flexibly enabled with fully leveraged 3D DRAM PNM bandwidth.
Yao-Lei Li, Wen-Bin Jia, Zhan-Chen Zhao et al.· International Symposium on L...· 0 citations
Diffusion models have shown marked advancements in 2-D generation and have also become focal points in 3-D generation via consistent multiview image generation. However, the computation and memory demands hinder their real-time deployment on mobile and edge devices. Moreover, the reduction of diffusion timesteps leads...
Wenxun Wang, Li-Kai Ma, Chen Tang et al.· IEEE Transactions on Compute...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.