Shi-Cheng Wei

Low-Rank Adaptation for Parameter-Efficient Fine-Tuning of Large Language Models

This review covers LoRA and its main variants and pays particular attention to the linear algebra behind them, and compares the major variants: quantized LoRA (QLoRA), quantization-aware LoRA (QA-LoRA), adaptive low-rank adaptation (AdaLoRA), sparse low-rank adaptation (SoRA), and weight-decomposed low-rank adaptation (DoRA).