Jul 2026
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD). However, full-vocabulary OPD typically assumes a shared tokenizer, while existing cross-tokenizer methods may discard teacher probability ma...
Hao Wang, Kun Yuan, Wenlin Zhong et al.
· arXiv.org · 5 citations
· ⚡1