Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation
On a 740-instance healthcare API routing task with a 1.5B Qwen student and a 20B teacher, eight KD variants are compared against supervised cross-entropy, finding single-seed evaluation is unable to detect central failure modes in small-model KD.
Dipto Sumit, Sakib Ul Haque, Farig Sadeque
· 0 citations