Privileged Context as Drift in On-Policy Self-Distillation
On-policy self-distillation (OPSD) trains a language model to match a copy of itself conditioned on privileged context. Existing work varies what privileged context contains and how it is produced while also changing models, data, and training setups, making the effects of privileged context design difficult to isolate...
Ravenor Davion, Nick Rui
· 0 citations