Function Over Form: Distributional Orthogonalization in Mixture-of-Experts with Replica Expert Mechanism
The scaling of LLMs increasingly relies on MoE architectures to decouple active computation from total parameter count. However, the efficacy of MoE is often constrained by expert collapse and representation redundancy, both leading to underutilization of model capacity. To address these challenges, this paper proposes...
Jin-Fan He, Yun-Zhuo Liu, Kai Zhang et al.
· 0 citations