Sensitivity-Guided Mixed-Precision Post-Training Quantization for MambaVision

Abstract

Hybrid vision backbones such as MambaVision combine convolutional layers, Mamba blocks (based on selective state space models), and self-attention within a single architecture. However, the heterogeneous operator composition of such models poses new challenges for post-training quantization (PTQ): uniform bit-width assignment causes catastrophic accuracy collapse due to widely varying per-block quantization sensitivity. In this work, we present a systematic per-block sensitivity analysis of MambaVision, revealing that mixer blocks (Mamba and attention) are remarkably robust to quantization down to 4-bit weights and 4-bit activations (average accuracy drop of only $\text{0. 0 6 \%})$, while a small number of bottleneck blocks (stem and downsampling layers) are extremely fragile under the same conditions. Based on these findings, we propose a sensitivity-guided mixed-precision assignment strategy that keeps critical blocks at 16-bit floating point, quantizes robust mixer blocks to 4-bit weights and 4-bit activations, and applies 8-bit weights and 8-bit activations to moderately sensitive blocks. Our best configuration achieves 80.66% Top-1 accuracy on ImageNet-1K with GPTQ (-3.29% from 16-bit floating point), recovering +4.68% over uniform 8-bit quantization at a comparable average bit-width of 8.14 bits, while reducing model size by 53.3% and bit operations by 70.8%. We further provide a Hessian mismatch analysis explaining the counterintuitive finding that GPTQ underperforms round-to-nearest under uniform quantization but recovers its advantage under mixed-precision, where 16-bit floating point bottleneck blocks act as error propagation firewalls.