Entropy Interventions Do Not Transfer: Why RLVR Entropy Control Misfires in On-Policy Distillation
Reinforcement learning with verifiable rewards (RLVR) controls policy entropy by reweighting tokens against an external scalar advantage. On-policy distillation (OPD) faces the same surface problem, a student that can sharpen too fast, but it replaces that advantage with a per-token teacher target. That shared symptom...