FedLSA: Cosine Triggered Reparameterization Augmented by Output Boundary Calibration for Private Federated Low Rank Adaptation
TL;DR
This work proposes FedLSA, a Federated Low-rank Subspace Adaptive update framework, which utilizes the cosine similarity of updates to the low rank coefficient matrix B as an economical proxy for subspace drift, which minimizes overhead on the server through dynamic Singular Value Decomposition (SVD) allocation.
Abstract
Low-Rank Adaptation (LoRA) has become the standard paradigm for Parameter-Efficient Fine-Tuning (PEFT) within federated learning (FL) under privacy preservation. However, while reparameterization strategies in recent orthogonalization baselines like FedSVD enhance matrix expressiveness, they often overlook the actual dynamic requirements for subspace updates during train-ing. To balance computational efficiency and model expressiveness under dif-ferential privacy (DP) constraints, we propose FedLSA, a Federated Low-rank Subspace Adaptive update framework. Specifically, FedLSA utilizes the cosine similarity of updates to the low rank coefficient matrix B as an economical proxy for subspace drift, which minimizes overhead on the server through dynamic Singular Value Decomposition (SVD) allocation. To complement this mechanism triggered by events, we establish an Output Boundary Calibration (OBC) within the logits space to ensure the optimization trajectory remains robust even when matrix basis updates are suspended. Experimental results on five representative Natural Language Understanding (NLU) benchmarks demonstrate that FedLSA consistently outperforms all evaluated baselines in terms of accuracy. Notably, compared to the strongest state-of-the-art (SOTA) baseline, FedLSA establishes a better Pareto front for efficiency and accuracy.