GSLoRa: Gradient spectral alignment for low-rank adaptation.
TL;DR
This work develops a low-rank adaptation framework based on gradient spectral alignment that achieves performance comparable to full fine-tuning while training only 0.1% of parameters, and provides novel insights into improving the efficiency, stability, and adaptability of PEFT frameworks.
Abstract
Parameter-efficient fine-tuning (PEFT) is a key technique for adapting large pre-trained language models to downstream tasks with minimal parameter updates. However, existing PEFT methods often suffer from slow convergence, gradient noise, and weak alignment between learned features and task semantics. To this end, we develop a low-rank adaptation framework based on gradient spectral alignment. Specifically, we (1) perform spectral decomposition of the pre-trained gradient covariance matrix and apply eigenvalue scaling to retain 95% of the spectral energy, optimizing parameter initialization; (2) design a dual gradient projector that combines orthogonal and spherical projections to suppress gradient noise, reducing its variance to 38.2% of standard LoRA; and (3) introduce a dynamic eigenvalue scaling mechanism that adaptively recalibrates principal component weights via a nonlinear scaling function, enhancing representational capacity. Experiments on the GLUE benchmark show that our method achieves performance comparable to full fine-tuning while training only 0.1% of parameters, outperforming AdaLoRA and DoRA by 6.2% and 5.3%, respectively. Additionally, our approach improves the mathematical reasoning accuracy of LLaMA-7B on GSM8K by 7.8%. This work provides novel insights into improving the efficiency, stability, and adaptability of PEFT frameworks. The code is available at https://github.com/rainylover/GSLoRA.