Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers. However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align. Existing logits-based methods are easy to...
TRACK: TRajectory-Aware Capacity routing via top-K selection is presented, a heterogeneous denoising strategy that switches between compatible large and small models at selected steps, reducing the average cost per denoising evaluation.
Mustafa Munir, Huy Vu, Shreyas Misra et al.· 0 citations
This work adapt and enhance preconditioned gradient methods to overcome the practical challenges of large-scale LLM pretraining and proposes algorithmic modifications including per-step QR orthogonalization and improved preconditioning strategies to enable stable training in these regimes.
Mikail Khona, Aditya Vavre, Boxiang Wang et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.