Latent-MOPD: Latent Multi-Teacher On-Policy Distillation
On-policy distillation (OPD) trains a student on the responses it generates. Existing LLM multi-teacher OPD transfers what specialists predict through their output distributions. We introduce Latent-MOPD, to our knowledge the first representation-level multi-teacher OPD method for LLMs. It integrates existing specialis...
Zheng-Yu Fang, Seoyeon Hong, Jie Yang et al.
· 0 citations