Preprint
Nov 2026
Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens
On-policy distillation (OPD) trains a student language model on its own trajectories using token-level supervision from a stronger teacher. We show that standard sampled reverse-KL OPD systematically suppresses updates to high-confidence tokens, such that their weak gradient contribution does not necessarily imply weak...
Ming-Yu Cao, Yao-Kai Jin, Shi-Wei Liu et al.
· 0 citations