HeimdaLLM: Efficient Cloud-assisted Federated Fine-tuning with Zeroth-Order Rectification for LLMs

Abstract

Large Language Models (LLMs) have achieved remarkable success in NLP tasks, but fine-tuning them on resource-constrained mobile devices remains challenging due to prohibitive memory and computation requirements. Federated Learning (FL) enables privacy-preserving distributed fine-tuning, yet conventional approaches, including full fine-tuning and parameter-efficient fine-tuning (PEFT), either rely on backpropagation, which is memory-intensive, or zeroth-order optimization (ZOO), which suffers from high variance and directional bias, leading to slow convergence and degrading accuracy. To address these limitations, we propose HeimdaLLM, a cloud-assisted federated fine-tuning framework that combines ZOO with Gradient Rectification (ZGR). In HeimdaLLM, FL clients perform ZOO updates on local domain datasets, while the assisted cloud conducts BP-based fine-tuning on datasets from the same domain and periodically transmits guided gradient direction (GGD) data to rectify ZOO process. To reduce client memory usage, computation, and transmission overhead, we introduce two system-level controllers: the System-level Pipeline Controller (SPC) and Data Transmission Controller (DTC), which pipeline model layer-wise transmission and computation of cloud and clients, and adaptively compress GGD data. Experiments on multiple models and NLP datasets show that HeimdaLLM reduces memory footprint for client devices by up to 86.37%, achieves up to 8.8× faster convergence than the baselines, and improves accuracy by up to 10% over state-of-the-art ZOO methods.