Counterfactual Relevance for On-Policy Distillation is introduced, which operationalizes task relevance through a paraphrase-calibrated counterfactual sensitivity margin and establishes CROP as a model-internal, contrast-specific method for allocating token-level supervision.
Abstract
On-policy distillation (OPD) supervises a student language model on trajectories sampled from its current policy, but assigns equal credit to response tokens with unequal supervision value. Selective OPD addresses this limitation by allocating supervision non-uniformly across response tokens according to their estimated training value. Most existing criteria, however, focus primarily on optimization need, such as uncertainty or teacher-student disagreement, while task relevance, namely whether the supervision is tied to the semantic content of the current input, remains less directly characterized as a complementary dimension. To address this gap, we introduce Counterfactual Relevance for On-Policy Distillation (CROP), which operationalizes task relevance through a paraphrase-calibrated counterfactual sensitivity margin. For each source prompt, CROP constructs a validated original-paraphrase-counterfactual triplet, holds the student rollout fixed, and measures each response position by its sensitivity to a task-relevant condition change calibrated by its sensitivity to a meaning-preserving rewrite. Matched selection controls show that CROP identifies more useful supervision positions than random or lowest-relevance selection, while component comparisons confirm the value of both counterfactual sensitivity and paraphrase calibration. Across two teacher-student settings, CROP improves aggregate performance by 1.92 and 2.96 points over the strongest non-CROP selector. These results support task relevance as a complementary criterion for selective OPD and establish CROP as a model-internal, contrast-specific method for allocating token-level supervision.
A unified view is developed in which the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization, which enables any-to-any transfer across models that differ in scale, training stage or generation.
Rong Wu, Dao-Cheng Fu, Yu Yang et al.· 0 citations
On-policy post-training can improve large language models by learning from their own trajectories, but requires costly rollout generation. We ask whether fixed demonstrations can support competitive off-policy learning through better supervision. Our premise is that their usefulness depends not only on the training tra...
Tian-Le Wang, Jia-Yu Liu, Rui-Zhi Zhao et al.· 0 citations
This work identifies Teacher Uncertainty Contraction (TUC), a systematic phenomenon whereby the teacher's predictive uncertainty decreases as it continues from a student-generated prefix, and proposes Adaptive-Continuations On-Policy Distillation (AC-OPD), which augments informative states along student rollouts with t...
Jin-Gang Zhou, Yu-Yi Zhou, Hai-Yang Guo et al.· 3 citations
Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.
Yang Li, Semih Yavuz, Shafiq Joty· 5 citations· ⚡1
DiffGate is introduced, an outcome-gated objective that combines GRPO with selective, bounded teacher guidance, and improves pass@8 across all four model--domain settings, demonstrating improved solution coverage under the evaluation protocol.
Karn Tiwari, V. Chordia, P. PrathoshA· 0 citations
Group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance, and demonstrates that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance.
Zhu Zhang, Ji-Xun Wang, Xiao-An Xu et al.· 3 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.