HeteroFL-LoRA: Federated LoRA Fine-Tuning Across Heterogeneous LFMs via Singular Value Collaboration

Abstract

Recent studies have integrated Low-Rank Adaptation (LoRA) into federated learning to enable parameter-efficient fine-tuning of Large Foundation Models (LFMs) across data silos, supporting responsible and privacy-preserving collaboration in distributed environments. However, these studies assume that all clients share the same LFM, so each client fine-tunes LoRA on private data using identical pre-trained weights and the server directly aggregates the resulting LoRA updates for collaborative training. The ideal assumption rarely holds in real-world deployments, where clients typically possess heterogeneous base LFMs. Because LoRA updates are anchored to model-specific pre-trained weights, direct aggregation is not only invalid but often counterproductive, leading to sharply degraded collaborative performance. To address this issue, we propose HeteroFL-LoRA, a federated framework that enables LoRA fine-tuning across heterogeneous LFMs. We express LoRA updates through singular value adjustments, activating task-specific knowledge in an parameter-efficient manner while decoupling them from model-specific pre-trained weights. Then, we introduce a Singular Matrix–Guided Subspace Projection that employs singular matrices to achieve cross-subspace mapping, enabling heterogeneous clients to aggregate their LoRA updates in a unified representation space. In this way, task-specific knowledge can be effectively transferred and shared across different LFMs. Experimental results demonstrate that HeteroFL-LoRA achieves effective and robust collaboration across heterogeneous LFMs while maintaining advantages in both training and communication efficiency.