DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation
Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA vari...
Yang-Ming Zhang, Ling Yi, Da-Tong Li et al.
· 0 citations