This paper builds on the singular value decomposition factorization of adapters to develop a framework based on Stein variational gradient descent (SVGD), which delivers strong model calibration and attains higher prediction accuracy than SVGD and related uncertainty estimation methods that are formulated in Euclidean space.
Abstract
Several geometry-aware approaches to low-rank adaptation have emerged for parameter-efficient fine-tuning of large pre-trained models. These methods aim to take full advantage of the geometric structure of low-rank manifolds for improving the efficiency in subspace utilization and reducing redundancy by enforcing orthogonality constraints during optimization. The strong empirical results of these techniques have motivated further study into whether predictions from such geometry-based adaptation methods could be overconfident. In this paper, we build on the singular value decomposition factorization of adapters to develop a framework based on Stein variational gradient descent (SVGD). In this formulation, the low-rank matrices are transported along the Stiefel manifold to match the targeted distributions while retaining their crucial geometric structure. Since this geometry-aware SVGD approach provides multiple solutions during inference, it supports uncertainty quantification and produces better-calibrated adapters on the Stiefel manifold. Extensive experiments show that our method delivers strong model calibration and attains higher prediction accuracy than SVGD and related uncertainty estimation methods that are formulated in Euclidean space.
SPIRA is a PEFT framework that separates support discovery from parameter-efficient adaptation and remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.
Zhong-Yi Wen, Zhikai Zhai, Guo-Min Sun et al.· IEEE Transactions on Pattern...· 0 citations
Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning, both motivated by robustness to perturbations. However, their relationship has remained largely unexplored. Building on classical second-order expansions of $f$-divergences,...
Zero-Training Fisher Geometry Alignment (ZFGA), a closed-form method that improves robustness under covariate shift without modifying model parameters, is proposed, offering a lightweight and deterministic alternative to optimization-based test-time adaptation.
Behraj Khan, T. Syed, Syed Ahmad Chan Bukhari· arXiv.org· 0 citations
FITR is proposed, a trust-region-based BO method that replaces lengthscale-based scaling by local pullback-Fisher weights and easily generalizes to non-isotropic surrogates, although the gains are more task-dependent in that setting.
Saksham Kiroriwal, Julius Pfrommer, Jürgen Beyerer· 0 citations
This paper derives the essential Riemannian tools and proposes efficient algorithms for their implementation of orthogonal two-factor matrices -- a flexible family of structured matrices formed by block-diagonal factors interleaved with fixed permutations.
Aliev S. E. Aliev, Maxim V. Rakhuba· arXiv.org· 0 citations
This work introduces a Nested Inductive Bias framework that utilizes a two-stage diffeomorphic composition to formally pull back non-Euclidean target geometries onto the SPD manifold, and proposes the Rational Conformal Metric (RCM), designed to establish state-of-the-art geometric robustness against outliers by boundi...
Tushar Das· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026