Model soup, averaging the weights of multiple fine-tuned models, delivers ensemble-level accuracy at single-model inference cost, but its success requires both linear mode connectivity (LMC) and sufficient diversity among candidates. We study these two requirements under quantization-aware training (QAT). First, we sho...
Jinwook Chung, Sungyeop Jung, Weronika Czorapinska et al.· Proceedings of the 32nd ACM...· 0 citations
SQuaT (Student-Aware Quantized Teacher Features), a label-free QAT framework with KD that theoretically eliminates this lower bound on the distillation loss by applying the student's quantization parameters to quantize the teacher's features during distillation is proposed.
H. Lee, Hyeonsik Jo, Jinwook Chung et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.