Preprint
Aug 2026
The Distributional View of Knowledge Distillation
Token-level knowledge distillation (KD) matches two conditional distributions per position, yet the standard objectives compare them pointwise: a Kullback-Leibler gradient is blind to which wrong token receives probability mass. We develop a distributional view in which the teacher is represented not by a single soften...
G. Verbii, Ju-Ho Lee
· 0 citations