Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning
On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation, shows that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.
Safaeid Hossain Arib, Rabeya Akter, I. N. Swapnil et al.
· 0 citations