Sep 2026· Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence· pp. 6027-6035· 0 citations· 37 references
TL;DR
This work proposes SeMi-LoRA (Separation and Mixing LoRA), a novel framework that enables complete high-rank updates while preserving mergeability and consistently outperforms strong PEFT baselines with favorable parameter efficiency.
Abstract
Low-Rank Adaptation (LoRA) has become a standard paradigm for parameter-efficient fine-tuning (PEFT). However, its low-rank constraint can limit its capacity to express complex weight updates, leaving a performance gap compared with full fine-tuning. Existing extensions improve expressivity, but they often sacrifice parameter mergeability or rely on sparse, block-diagonal updates that restrict global information flow. We propose SeMi-LoRA (Separation and Mixing LoRA), a novel framework that enables complete high-rank updates while preserving mergeability. SeMi-LoRA decomposes adaptation into channel-wise compression followed by two-stage latent mixing, namely Rank Mix and Channel Mix. Our structural analysis shows that, under a fixed base rank, the rank capacity of the update increases with the number of channels and supports complete updates rather than partial sparse ones. Extensive experiments on commonsense reasoning, mathematical reasoning, natural language understanding, and dialogue generation benchmarks demonstrate that SeMi-LoRA consistently outperforms strong PEFT baselines with favorable parameter efficiency.
Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead. Across diverse and challenging fine-tuning tasks, TaRA consistently outperforms prior state-of-the-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization.
This paper introduces a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead, and employs a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients.
This review covers LoRA and its main variants and pays particular attention to the linear algebra behind them, and compares the major variants: quantized LoRA (QLoRA), quantization-aware LoRA (QA-LoRA), adaptive low-rank adaptation (AdaLoRA), sparse low-rank adaptation (SoRA), and weight-decomposed low-rank adaptation (DoRA).
Shi-Cheng Wei· Theoretical and Natural Scie...· 0 citations
This work reformulates LoRA-based CL as a consistent feature mapping problem that mimics the behavior of the joint-training upper bound, wherein a unified adaptation parameter matrix is learned to simultaneously capture the input-output relationships established by all task-specific LoRAs.
Yue Lu, Shizhou Zhang, De Cheng et al.· IEEE Transactions on Pattern...· 0 citations
Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representational capacity, the optimizer shapes how much of that capacity is used in the induced weight-space updates. In a case study of GPT-2 adaptation with LoRA, we observe a strong rank-dependent optimizer effect. Despite using the same nominal rank, AdamW often produces per-step updates with concentrated singular spectra and low effective rank, whereas Muon uses a richer set of directions and benefits more consistently from increasing LoRA rank. These observations motivate ISO-LoRA, an optimizer that couples the LoRA factor updates through spectral descent on the induced tangent perturbation in weight space. ISO-LoRA promotes updates that distribute energy more evenly across singular directions, improving rank utilization while preserving compatibility with the LoRA parameterization. We complement this design with theoretical guarantees showing that ISO-LoRA can achieve higher effective rank than standard factor-wise optimizers through a one-step analysis under a stylized spiked-gradient model. We validate this design on language-model adaptation across 0.1B-7B-parameter models, where ISO-LoRA improves effective rank and downstream performance, with the strongest gains at moderate-to-large LoRA ranks. Our results highlight rank utilization as a key factor in LoRA optimization and suggest that optimizer design offers an important path toward stronger parameter-efficient adaptation.
Zi-Han Zhu, Zhe-Hang Du, Xu-Yang Chen et al.· 0 citations
Low-Rank Adaptation (LoRA) has become a widely adopted technique for efficient neural network fine-tuning, decomposing model updates into low-rank matrices. However, LoRA remains computationally costly because it updates all matrices uniformly, regardless of their actual contribution to adaptation. This cost is especially prohibitive for large-scale models with billions of parameters and for resource-constrained settings such as edge deployment and on-device fine-tuning. We show for the first time that not all LoRA matrices are equally worth tuning: matrices with smaller condition numbers (the ratio of largest to smallest singular value) are already well-balanced across directions and contribute only marginally to adaptation, whereas matrices with larger condition numbers contain underdeveloped directions that span richer subspaces and drive most of the performance gains. This observation itself is a key contribution of our work, and it motivates a more selective approach to fine-tuning. Building on this insight, we propose \k{appa}-LoRA, a method that optimizes LoRA by focusing updates on the matrices with the largest condition numbers, which capture the most informative directions of change. By restricting LoRA updates to the top 50% of weight matrices ranked by condition number, \k{appa}-LoRA halves the trainable parameter count and correspondingly reduces compute and memory cost. Extensive experiments across multiple benchmarks show that this design cuts fine-tuning time by 16.2% on average while matching the accuracy of standard LoRA and reducing memory cost by 4.5%. Further analysis reveals that the condition numbers of the selected matrices consistently decrease over training, suggesting that \k{appa}-LoRA's effectiveness stems from targeted spectral rebalancing rather than parameter selection alone.
Jianghui Wang, Si-Long Yong, Francesco Orabona et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
