Skip to content

Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillation

Sep 2026 · arXiv.org · Vol abs/2609.39120 · 0 citations · 53 references
Computer Science

TL;DR

S-OPD is a simple multimodal on-policy distillation framework that explicitly strengthens student perceptual learning through two objectives: Teacher-calibrated Policy Contrast separates student policies under original and masked images with teacher-based token-level gating, strengthening the student's reliance on visual evidence during reasoning.

Abstract

On-policy distillation (OPD) improves reasoning by providing token-level supervision from a teacher on a student's own trajectories. Existing methods primarily focus on enhancing this teacher-side guidance (e.g., by enriching teacher inputs and refining teacher feedback), yet we find that limited student perception is another critical bottleneck in multimodal OPD. By providing oracle visual facts, the performance of OPD-trained students can still be substantially improved for both weak and strong teachers. To address this bottleneck, we propose S-OPD, a simple multimodal on-policy distillation framework that explicitly strengthens student perceptual learning through two objectives. Specifically, Teacher-calibrated Policy Contrast separates student policies under original and masked images with teacher-based token-level gating, strengthening the student's reliance on visual evidence during reasoning. Policy Agreement aligns student policies under original and noise-perturbed images, further improving perceptual robustness to visual noise. Notably, our method can be seamlessly plugged into existing OPD frameworks, requiring no additional data annotations, model parameters or inference operations. Extensive experiments on eight benchmarks across student scales and distillation paradigms demonstrate consistent performance improvements, with gains of up to 4.25 points on LogicVista. When combined with existing teacher-side supervision methods, our method can yield further gains. Code is available at https://github.com/Sirilaw/S-OPD.

View source

Similar papers

#machine learning Preprint Nov 2026

Reinforcing What the Student Has Learned: Teacher-Guided On-Policy Distillation

Student-Delta On-Policy Distillation (SD-OPD), which uses centered student--base differences as update coefficients, keeps each coefficient only when its sign agrees with centered teacher--student feedback, and recomputes the student signal every training round.

Bo-Zhen Fang, Lin-Hao Zhong, Xue-Hao Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers

Reinforcement learning can substantially improve a reasoning teacher, but it is unclear which of those improvements survive when the teacher supervises a smaller on-policy student. We study this question in mathematical reasoning by comparing teacher lineages before and after GRPO, multiple student scales, direct GRPO,...

Xiao-Yu Chen, Bo Shao, Tian-Gang Zhu et al. · 0 citations
#machine learning Preprint Oct 2026

How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation

On-policy distillation (OPD) improves the reasoning capabilities of small language models through token-level teacher supervision on student-generated trajectories. Yet can teachers that excel at solving problems independently also guide student reasoning effectively? Prior work shows that when student prefixes follow...

Xiao-Yu Ma, Hao-Yue Liu, Zhi-Chao Wang et al. · 0 citations
#machine learning Preprint Oct 2026

Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models

Improving the reasoning capabilities of vision-language models (VLMs) often relies on post-training with supervision from stronger models. Two common strategies are supervised fine-tuning (SFT), which trains on teacher-generated demonstrations, and on-policy distillation (OPD), which provides teacher feedback on trajec...

Frank Yang, Kefei Yan, Yi-Ren Wang et al. · 0 citations
Preprint Sep 2026

Train4Merge: RL or SFT---Which Yields Better Teachers for OPD-Based Model Merging?

Domain experts trained from a shared checkpoint can transfer their specialized capabilities to a single student through on-policy distillation (OPD). Existing research primarily focuses on improving this merging process, while the algorithms used to train the experts have received limited systematic comparison. We inve...

Jing-Yuan Huang, Zu-Ming Huang, Yu-Cheng Shi et al. · 0 citations
Preprint Oct 2026

OPDArena: Understanding Learning and Forgetting in On-Policy Distillation

On-policy distillation (OPD) is increasingly used for language model post-training, yet how teacher supervision shapes learning and forgetting remains unclear. We present OPDArena, a framework comparing supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and OPD across five pretrained m...

Yuan Xu, Tian-Zhu Ye, Hui Sun et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.