Condition-Contrastive Extrapolation for On-Policy Distillation of Language Models
On-policy distillation (OPD) improves a student language model by training on its own generated trajectories with supervision from a stronger teacher. However, when conditioned on student-generated prefixes, the teacher can favor invalid continuations that violate the question's premises, causing OPD to encourage erron...
Wen-Xuan Luo, Chen-Xi Li, Hao-Yue Liu et al.
· 0 citations