Data-Adaptive Lower-Rank Adaptation (DALorRA), a simple and effective variational Bayesian sparse framework that shifts the paradigm of uncertainty quantification from the dense parameter space to the lightweight rank level of low-rank adaptation (LoRA).
Abstract
Large language models (LLMs) exhibit remarkable reasoning capabilities, but their task-specific fine-tuning is notoriously plagued by overconfidence, severely hindering trustworthy deployment. We propose Data-Adaptive Lower-Rank Adaptation (DALorRA), a simple and effective variational Bayesian sparse framework that shifts the paradigm of uncertainty quantification from the dense parameter space to the lightweight rank level of low-rank adaptation (LoRA). With the insight that LoRA essentially aggregates multiple rank-one components that may provide superfluous model capacity, DALorRA imposes stochastic masking on rank dimensions, enabling Bayesian regularization of model capacity during training and ensemble-like calibration during inference. Extensive experiments demonstrate DALorRA's excellent calibration of LLMs without compromising reasoning accuracy.
This paper investigates SoRA algorithm's performance in large-scale models, and derives an improved learning rate configuration and proposes SoRA+, an enhanced variant that introduces a hyperparameter to adjust learning rates proportionally and mitigates the suboptimality of the original SoRA method.
This work proposes SeMi-LoRA (Separation and Mixing LoRA), a novel framework that enables complete high-rank updates while preserving mergeability and consistently outperforms strong PEFT baselines with favorable parameter efficiency.
Zhen-Fei Yang, Beiming Yu, Peiqin Lin et al.· Proceedings of the Thirty-Fi...· 0 citations
Layer-wise Distribution Alignment (LDA), a lightweight inference-time correction that uses layer-wise calibration statistics to align reduced-routing representations with the default configuration, is proposed, recovering much of the performance lost induced by the distributional shift under reduced routing while prese...
Dohyeon Kim, Bedionita Soro, Sung Ju Hwang· 0 citations
Reinforcement learning (RL)-based post-training has become an effective approach for eliciting reasoning capabilities in large language models (LLMs). However, adapting post-RL models to new knowledge domains or behaviors through subsequent supervised fine-tuning (SFT) can severely overwrite these capabilities. Existin...
Wen-Zhi Fang, Nicholas Tzou, L. Valkov et al.· 0 citations
Low-Rank Adaptation (LoRA) achieves parameter-efficient fine-tuning by constraining model updates to a low-rank subspace and has been widely used in practice. However, LoRA typically employs a shared low-rank update across tokens, which limits its ability to fully exploit the adaptation subspace for tokens from differe...
Guang Yang, Chang-Hao Guan, Chao Huang et al.· 0 citations
Extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.
Sankar Behera, Dhruv Singh, Anshika Agnihotri et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
