Skip to content

Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation

Sep 2026 · arXiv.org · Vol abs/2609.34738 · 1 citation · 31 references
Computer Science

TL;DR

ESCD aggregates prefix-related teacher events and supervises the total probability of byte-compatible one-step student completions, avoiding tokenizer-dependent probability splits among individual tokens, and support event entry and event completion as complementary supervision targets for cross-tokenizer knowledge transfer.

Abstract

On-policy distillation (OPD) transfers knowledge between language models through teacher supervision on student-generated trajectories. With different tokenizers, a single teacher token may require multiple student tokens to generate, creating intermediate states where the event is entered but not yet completed. Existing cross-tokenizer methods align tokens or text spans to construct comparable prediction targets. We study a complementary problem after partial generation: once the student produces a prefix of a teacher token, multiple next tokens may complete the same remaining bytes, but the teacher only specifies the required completion rather than how probability should be divided among these valid continuations. We introduce Event-Set Completion Distillation (ESCD), which complements cross-tokenizer probability alignment with completion-set supervision. ESCD aggregates prefix-related teacher events and supervises the total probability of byte-compatible one-step student completions, avoiding tokenizer-dependent probability splits among individual tokens. The method reuses student trajectories and predictions, requiring neither additional rollouts nor changes to the student vocabulary. Experiments demonstrate consistent gains in mathematics, code, and scientific reasoning across model families and tokenizers, extending to large-scale MoE distillation from a 1T teacher to a 35B student. Local analyses show that retaining completion sets better matches the reference supervision, while one-step completion covers over 99% of observed compatible teacher mass after partial event entry in the studied tokenizer pairs. These findings support event entry and event completion as complementary supervision targets for cross-tokenizer knowledge transfer. Code will be released on GitHub.

View source

Similar papers

Dual-Vocabulary Language Model for Cross-Tokenizer Distillation

On-policy distillation (OPD) bridges teacher supervision and student behavior, but different teacher-student tokenizers introduce misalignment in both input tokenization (#1) and output logits (#2). Existing approaches address the former by matching same-text spans or converting tokens to bytes, often losing fine-grain...

Ke-Di Chen, Chen Lin, Yutao Sun et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability

Compact supervision at strict positions can be more effective than broader coverage that introduces weakly aligned or conflicting training signals: compact supervision at strict positions can be more effective than broader coverage that introduces weakly aligned or conflicting training signals.

Bing-Xi Hou, Guochao Jiang, Guo-Feng Quan et al. · 0 citations
#machine learning Preprint Oct 2026

CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation

CAST (Chunk-aware credit Assignment with Selective Token Anchoring) redistributes each chunk's advantage across student tokens using predictive uncertainty and local teacher-signal smoothness, emphasizing learning opportunities supported by locally consistent teacher feedback while preserving the chunk's total advantag...

Jing-Ru Yang, Jin-Hang Wu, Min-Jing Shi et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which promp...

Xiang-Yuan Chen, Fu-Tao Su, Kong Wang et al. · 0 citations
Preprint Oct 2026

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers. However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align. Existing logits-based methods are easy to...

Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge et al. · 0 citations
#natural language process... Preprint Oct 2026

PA1-OPD: One-Step Prefix-Augmented On-Policy Distillation for Language Model Pretraining

Knowledge distillation (KD) improves language-model pretraining by matching teacher distributions at corpus prefixes, but provides no direct supervision at contexts induced by student predictions. We introduce PA1-OPD, which augments corpus-path distillation with teacher feedback after one student-sampled transition pe...

Zhen-He Wu, Jian Yang, Ya-Ping Jin et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.