Activation-Conditioned Self-Distillation
On-policy self-distillation uses a model as its own teacher to provide dense supervision for reasoning, often through reference-solution conditioning. Providing privileged information does not by itself ensure effective token-level supervision throughout long responses. We introduce Activation-Conditioned Self-Distilla...
Zhe-Xi Lu, Subhajit Chaudhury, Tejaswini Pedapati et al.
· 0 citations