Q-SA2: Enhanced low-bit quantization-aware fine-tuning via structure-aware adaptation.

Abstract

Quantization-Aware Fine-tuning (QAF) is critical for deploying large language models in resource-constrained scenarios. While in low-bit QAF, strong downstream performance hinges on both minimizing model quantization errors and providing robust initialization of low-rank adaptation. To further improve the model performance, we propose Structure-Aware Adaptation for low-bit QAF (Q-SA2) in this paper. By decomposing pre-trained weights into sparse components and principal subspaces, Q-SA2 provides a favorable initialization for downstream task adaptation. Specifically, to better preserve the representations of pre-trained weights and minimize errors, the sparse components are quantized, and the low-rank adapters are initialized from the principal subspaces. Then, we extend this framework to a weight-wise mixed-precision variant with a distribution-based outlier detection strategy on sparse components, enabling Q-SA2 to adaptively allocate higher precision to critical weights. Comprehensive experiments on WikiText-2, C4, GSM8K, and the GLUE benchmark demonstrate that the proposed method consistently outperforms state-of-the-art QAF methods, particularly on mathematical reasoning tasks, achieving an average improvement of 4% under 2-bit quantization and up to 7.78% on the Qwen2.5-7B model under 4-bit quantization. The code is available at https://github.com/Rainiay/Q-SA2.git.