Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution
We study post-training for dynamically routed mixture-of-experts (MoE) models, focusing on integrating independently optimized specialists while keeping consolidation efficient. Our approach combines calibrated multi-teacher on-policy distillation (C-MOPD), which matches each selected teacher on a local Top-$K$ support...
Yue He, Jia-Hui Hu, Sha-Man Tang et al.
· 0 citations