Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
The Contrastive Routing Mechanism (CoRM) is proposed, which scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference state, interpreted through a distinct per-expert projection.
N. Xiros, Dimitrios Damianos, Maria-Eleni Zoumpoulidi et al.
· 0 citations