Triplet Decomposition and Extensions: A General Framework for Parameter-Efficient Fine-Tuning.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods enable adapting large pre-trained models to downstream tasks with minimal overhead. Current approaches predominantly rely on low-rank decomposition to reparameterize weight increment matrices, assuming that model updates follow low-rank patterns. However, weight updates during fine-tuning may exhibit more complex statistical properties. An alternative paradigm decomposes weight increment matrices into frequency-domain components, offering potentially superior expressivity through flexible frequency component combinations. In this work, we unify these approaches under a Triplet Matrix Decomposition framework and rigorously compare their expressivities. Our analysis reveals that frequency-domain methods can surpass low-rank approaches when optimal frequency components are selected, and this advantage stems from orthogonal transformation matrices and flexible basis vector combinations. Building on these insights, we propose Learnable Orthogonal Adaptation (LoTA), a novel PEFT method that learns task-specific transformations and adaptive basis combinations. LoTA employs cascaded Householder transformations to construct orthogonal matrices with minimal parameters while ensuring exploration of the complete orthogonal space. We address the discrete optimization challenge of basis selection through finite-difference gradient approximation, enabling end-to-end backpropagation. Extensive experiments including natural language understanding, mathematical reasoning, commonsense reasoning, computer vision and visual instruction tuning tasks demonstrate that LoTA achieves superior parameter efficiency and performance compared to existing PEFT methods.