Federated Parameter-Efficient Selective Fine-Tuning for Vision-Language Models via Global-Local Masking
TL;DR
This paper proposes a personalized Federated Mask-based Selective PEFT framework, which significantly outperforms the existing federated PEFT method on both seen and unseen classes, and achieves balanced performance on all clients.
Abstract
While Parameter-Efficient Fine-Tuning (PEFT) with vision-language foundation models (VLMs) achieves strong in-domain performance in federated learning (FL), generalizing VLM-based FL methods to unseen domains remains challenging. To address this, this paper focuses on FL in base-to-new settings, examining two key trade-offs: (1) the performance trade-off between seen and unseen classes and (2) the performance trade-off among different clients. This paper explores selective mask-based PEFT, a technique rarely used for VLM-based FL, to address these two challenges, leveraging its ability to effectively utilize the prior knowledge of VLMs and its flexibility for client-specific adaptation. Specifically, we propose a personalized Federated Mask-based Selective PEFT framework (pFedMSF). To enhance generalization to unseen classes, pFedMSF employs binary masks to select a small subset of tunable weights, preserving the prior knowledge of the pre-trained VLM. To address the performance trade-off among clients, besides global masks shared by all clients, pFedMSF further introduces local mask for personalized model weight update to individual client. Experimental results show that pFedMSF significantly outperforms the existing federated PEFT method on both seen and unseen classes, and achieves balanced performance on all clients.
