A Riemannian Geometry for Low-rank Adaptation
Low-rank adaptation (LoRA) is widely used as a parameter-efficient fine-tuning technique for pre-trained deep neural networks, which approximates the weight update via full fine-tuning by a low-rank matrix $BA^\top$. This parameterization leads to the equivalence relation $(B, A) \sim (BG^{-1}, AG^\top)$ for any invert...
Shoichiro Takeda, Shin'ya Yamaguchi, Satoshi Suzuki et al.
· 0 citations