ReTaCo: Residual-Target Control for On-Policy Distillation
On-policy distillation (OPD) trains a student on its own generated prefixes with token-level teacher feedback, but transmitting or storing the teacher's full-vocabulary distribution at every token is costly. Entropy-aware OPD (EOPD) adds forward supervision to reverse KL to help the student recover plausible tokens it...
Zi-Xiang Ni, Zhuo-Lun Hu, Ren-Jie Cao et al.
· 0 citations