LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment
This paper introduces a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead, and employs a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients.