Skip to content
Log in
Preprint

A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs

Abhinand Balachandran Praveen Prashant
Aug 2026 · 0 citations · 12 references
Computer Science

Abstract

Knowledge distillation trains a smaller student to match the outputs of a larger teacher. Feature-based methods also align intermediate representations, but this extra constraint may affect students differently. We study this question on CIFAR-100 using a ResNet-50 teacher, a width-controlled CustomResNet family and MobileNetV2 as a cross-design comparison. For each student, we evaluate each feature method against a matched logit-KD run using the same teacher, optimizer settings, training schedule and seed. We repeat the main comparisons across multiple seeds. Logit KD improved every tested student over its scratch baseline. Attention Transfer showed no clear relationship with size inside the CustomResNet family, but its average effect was negative for that family and positive for MobileNetV2. FitNets was below logit KD in all 15 paired runs. Within the constant-depth width sweep, its gap increased for wider students, although the different-depth w=48 student did not follow this trend. Finally, the same auxiliary coefficient produced different gradient scales across students, showing that a fixed coefficient does not create a uniform training condition.

arXiv Semantic Scholar
View source

Similar papers

Preprint Jul 2026

Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations

Shift-Augmented Knowledge Distillation (SAKD) is proposed, a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation, enabling single-stage training while producing adaptive, diverse views through a parameter-free cyclic shift.

Xuyi Yu, Yaohua Liu, Chengjun Li et al. · 0 citations
Preprint Aug 2026

Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs

ADAPT---Amortized Distillation Across Post-Trained LLMs is introduced, a framework for amortizing distillation across both axes of a model family: size and post-training variant, producing models for interpolated sizes across post-trained variants with a single distillation run.

Yan Zhou, Sara Kangaslahti, Jonathan Geuter et al. · 0 citations
Preprint Jul 2026

SCOReD: Student-Aware CoT Optimization for Recommendation Distillation

Student-Aware CoT Optimization for Recommendation Distillation (SCOReD), a CoT optimization framework tailored to recommendation that first parses each teacher trace into typed segments and uses the student LLM's attention to score the importance of each segment.

H. S. Shahgir, Yufei Li, Xiaohan Wei et al. · 0 citations
Review 2025

How can machine learning personalize educational content generated by LLMs

It is suggested that ML-guided prompt routing can improve perceived relevance of instructional materials while remaining transparent and easy to calibrate and integrate interpretable ML predictions with prompt-engineered LLMs can automate individualized content generation and improve learning outcomes.

Petr Tsekoyev, T. Sembayev, Z. Nurbekova · 0 citations
Preprint Jul 2026

LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

Experiential Learning is proposed, which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach, and establishes experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.

Tianzhu Ye, Li Dong, Guanheng Chen et al. · 1 citation