AFLoRA: Adaptive LoRA for Resource-Efficient Federated Fine-Tuning of Large Language Models

Abstract

LoRA-based federated fine-tuning allows clients to fine-tune LoRA adapters locally and aggregates updates on the server, thereby enabling the efficient adaptation of global foundation models to distributed, domain-specific data. However, existing methods assume identical and static LoRA adapters across clients, which not only limits the utilization of heterogeneous resources and adaptation to diverse data distributions, but also prevents dynamic adjustment to clients’ evolving local fine-tuning state. They also struggle to achieve precise and lightweight aggregation of LoRA matrices, hindering learning performance in resource-constrained environments. To address these issues, we propose AFLoRA, an adaptive LoRA-based resource-efficient federated fine-tuning framework. For effectively adapting to Non-IID local data while minimizing resource consumption, AFLoRA introduces diagonal-based dynamic rank assignment, which dynamically adjusts local LoRA ranks based on client resources and local adaptation state guided by a learnable diagonal matrix. It further decouples LoRA matrices updates between clients and the server to reduce overhead, and develops a zero-padding-based rank-aware aggregation mechanism to accurately integrate heterogeneous updates through lossless matrix expansion and rank-aware weighting. These mechanisms enable AFLoRA to balance resource efficiency and fine-tuning performance effectively. Extensive experiments demonstrate that AFLoRA outperforms state-of-the-art methods in both LLM adaptation performance and fine-tuning efficiency in heterogeneous environments.