Selective Tensor Freezing for Efficient Fine-Tuning in Resource-Constrained Federated Learning

Abstract

Fine-tuning remains essential for adapting large models to diverse downstream tasks, yet doing so in a privacy-preserving and resource-efficient manner is challenging, particularly in federated learning (FL) on edge devices. Parameter tensor freezing is a promising solution. However, current methods face key limitations. Static tensor freezing struggles to adapt to the non-IID (non-independent and identically distributed) data distributions across FL clients, while localized dynamic freezing may lead to slow convergence or divergence across clients, harming overall accuracy. We propose FedFreeze, a communication-aware and dynamic tensor-freezing framework for federated fine-tuning over resource-constrained edge networks. FedFreeze delivers two key benefits: 1) it explicitly incorporates computation costs and bandwidth-dependent communication costs into freezing-mask optimization, actively selecting which tensors are updated and transmitted to reduce computation and communication overheads; and 2) it improves convergence stability by coordinating freezing decisions based on sampled client statistics. We further analyze the memory usage patterns of FedFreeze and introduce the first kind of memory management strategy to minimize memory consumption of tensor-freezing based methods in FL. Experimental results based on real-world traces from NVIDIA Jetson hardware demonstrate that FedFreeze accelerates convergence by up to $5.46\times $ and reduces peak memory usage by up to 53.9% without compromising model accuracy. Furthermore, evaluations under heterogeneous computation and communication environments confirm its robustness.