SDE : Scale-Difference Evolution Knowledge Distillation

Abstract

Logit-based knowledge distillation (KD) is pivotal for efficient model compression and cross-architecture learning. However, conventional methods typically rely on static, single-scale logit alignment, thereby overlooking the semantic evolution trajectory embedded in cross-scale prediction transitions. To bridge this gap, we propose Scale-Difference Evolution distillation (SDE), formulated in a structure-aware manner. Unlike the traditional prediction-imitation paradigm, SDE explicitly models the difference vectors between multi-scale logits to capture dynamic logical transitions from local features to global semantics. To optimize the distillation signal-to-noise ratio, SDE decouples scale differences into two complementary modules: Category-Dominant Difference (CDD), which isolates evidence fluctuations for core categories via a Top-K attention mask, and Relational Structure Difference (RSD), which preserves the consistency of inter-class topological evolution. Extensive experiments demonstrate that SDE consistently outperforms state-of-the-art methods across CIFAR-100, Tiny-ImageNet-200, CUB-200 and Stanford Cars. Notably, SDE achieves a significant 8.10% accuracy boost on the CUB-200 fine-grained benchmark, highlighting its superior capability in resolving inter-class ambiguities through cross-scale semantic modeling.