Knowledge Distillation Across Tasks and Model Families: A Comparative Simulation Study—Denoising, Dark Knowledge, and the Role of Model Capacity
A comparative simulation study of distillation in structured-data settings, where neural networks, tree ensembles, kernel methods, nearest-neighbor methods, and linear models are plausible competitors, shows that useful transfer requires teacher predictions that remain conditionally aligned with the inputs.