On-policy Verbal Distillation is introduced, a framework that uses verbal scores from black-box teachers to rank student-generated sub-trajectories, retaining high-scoring ones and replacing low-scoring ones with teacher-generated continuations and suggests that retaining student-generated prefixes helps preserve explo...
Jing Xiong, Hui Shen, Shansan Gong et al.· arXiv.org· 8 citations
PerSyn (Personalized data Synthesis), a novel synthesis strategy that operates under a new ``Route then Generate''paradigm to create data tailored to each student model, enabling it to learn more effectively, is proposed.
Hengyuan Zhang, Shi-Ping Yang, Xiao Liang et al.· Annual Meeting of the Associ...· 21 citations
Prefilling-dLLM is proposed, a training-free prefill-decode disaggregation framework for dLLMs that partitions the prefix into N chunks, caches their KV representations once, and selects the top-K most relevant chunks with intra-chunk token sparsity for decoding, showing that sparse prefilling can outperform dense atte...
Jingfei Xiong, Qilong Han, Shansan Gong et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.