When and How to Rotate: Loss Design and Activation Granularity for Rotation-Based Vision Model Quantization

Abstract

Orthogonal-transformation-based post-training quantization (PTQ) mitigates activation outliers in large language models (LLMs), but its use for vision models remains unclear: when learned rotations are necessary, which training objectives are stable, and how activation quantization granularity affects performance. We empirically characterize rotation-based PTQ on eight vision models (RT-DETR, RF-DETR, and Mask R-CNN for detection; DeiT-S, Swin-T, ViT-S, ResNet-50, and MobileNetV2 for classification) across rotation mode, sharing strategy, objective, and activation granularity, with three findings. First, the floating-point (FP) recovery of the quantization-only baseline (QO-FP recovery) is an empirical indicator of when learned rotation is beneficial, with architecture- and task-specific exceptions. A framework using setting-specific empirical thresholds is evaluated on twenty additional architectures, with 12 of 13 deterministic mode recommendations selecting the more accurate mode. Its original eight-model instantiation has a 1.07% average gap to the seed=42 within-search Oracle; the revised 50–90% Transformer branch remains uncertain. Second, the effective objective is task- and bit-width-dependent: final-output knowledge distillation (Final KD) exceeds task loss at every matched seed on four of five classification models and on average on MobileNetV2, whereas block-output mean squared error (O-MSE) exceeds task loss over three seeds on RT-DETR W4A4 ( $0.081\pm 0.005$ vs. $0.065\pm 0.006$ ). Third, switching from per-tensor to per-token activation quantization in W4A4 improves four classification models by +3.43 to +23.32 pp and detection mAP from 0.065 to 0.375 (RT-DETR) and from 0.166 to 0.388 (RF-DETR), whereas per-token trails per-tensor in ViG-S graph-based classification. Rotation, objective, and granularity must therefore be selected jointly.