Micro-expert pruning reduces the capacity of mixture-of-experts (MoE) models by removing coupled channels within routed experts. However, finer pruning granularity alone does not guarantee the preservation of downstream capabilities, as similar language-modeling perplexity can accompany substantially different task per...
Zu-Kang Xu, Xin-Pei Gao, Xing Hu et al.· 0 citations
Looped Language Models (LoopLMs) improve parameter efficiency by recursively reusing shared Transformer modules, yet their repeated computation over evolving recurrent states poses new challenges for low-bit deployment. Existing post-training quantization (PTQ) methods, largely designed for feed-forward Transformers, f...
Zhi-Xiong Zhao, Zu-Kang Xu, Xing Hu et al.· 1 citation
ACE is proposed, a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping in MoE-based LLMs that consistently outperforms existing static and dynamic baselines.
Zu-Kang Xu, Zhi-Xiong Zhao, Xing Hu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.