Rethinking Self-Distillation for Multi-Teacher Capability Merging
Combining capabilities of multiple expert models trained starting from the same base checkpoint has become increasingly common in frontier language-model post-training. Recent trends suggest that multi-teacher on-policy distillation (MOPD) outperforms conventional off-policy methods. However, despite the higher inferen...
Roy Xie, Dan Friedman, Feng Nan et al.
· 0 citations