2025· Neural Information Processing Systems· 0 citations· 31 references
Computer Science
TL;DR
FedRAM is proposed, a three-step framework that progressively updates two scalar hyperparameters: the task importance weight and the client aggregation coefficient, where the proxy model serves as an intermediate between the local reference model and the global agent model.
Abstract
Federated Multi-Task Learning (FL-MTL) enables clients with heterogeneous data to collaboratively train models capable of handling multiple downstream tasks. However, FL-MTL faces key challenges, including statistical heterogeneity, task interference, and the need to balance local learning with global knowledge sharing. Traditional methods like FedAvg struggle in such settings due to the lack of explicit mechanisms to address these issues. In this paper, we propose FedRAM, a three-step framework that progressively updates two scalar hyperparameters: the task importance weight and the client aggregation coefficient. FedRAM introduces a reference-proxy-agent strategy, where the proxy model serves as an intermediate between the local reference model and the global agent model. This design reduces the need for repeated local training while preserving local performance. Extensive experiments on six real-world FL-MTL benchmarks show that FedRAM improves performance by at least 3 % over the most baseline on both in-domain and out-of-domain tasks, while reducing computational cost by 15 × . These results make FedRAM a robust and practical solution for large-scale FL-MTL applications. The code is available at https://github.com/wwffvv/FedRAM .
Federated learning (FL) enables collaborative model training across distributed clients while keeping data local. A central challenge is determining which client updates are beneficial for aggregation with respect to each client's target domain. Existing methods typically address this problem in parameter space by comparing model parameters or gradients. However, parameter-space similarity can be a poor proxy for predictive behavior, especially under heterogeneous, non-IID data. Consequently, updates that are misaligned with a client's target domain, including those caused by heterogeneous data or malfunctioning clients, may degrade local model performance. We propose Local Inference Guided Aggregation for Heterogeneous Training Environments to Yield Enhancement Through Agreement and Regularization (LIGHTYEAR), a federated learning framework that performs update selection in function space. LIGHTYEAR uses an NTK-based agreement score to characterize predictive behavior and determine a personalized aggregation set for each client. By relating model parameters to local predictive responses, the Neural Tangent Kernel (NTK) provides a more expressive criterion for update selection than parameter-space similarity alone. Because function-space information is not available before aggregation in conventional centralized FL, LIGHTYEAR uses a peer-to-peer (P2P) topology in which clients exchange updates directly and evaluate incoming models on private validation data. Each client selects only updates that are beneficial for its own target domain and aggregates them using a regularized rule that improves stability under heterogeneity. Across five datasets and nine baseline methods, LIGHTYEAR consistently outperforms centralized FL baselines and existing P2P approaches.
Mirko Konstantin, S. Zachow, Anirban Mukhopadhyay· 0 citations
In agentic AI-enabled edge computing, decentralized federated learning (DFL) leverages peer-to-peer model aggregation to improve the performance of on-device large language models (LLMs) without introducing a single point of failure, thereby enhancing local agents’ capabilities for decision-making. To protect the right to be forgotten for each agent, as required by data regulations such as the General Data Protection Regulation (GDPR), federated unlearning aims to remove the influence of a target agent’s data from the trained LLM while preserving model utility. However, existing federated unlearning methods predominantly assume centralized architectures and face two critical challenges when extended to decentralized federated learning systems: 1) requiring all remaining agents to participate in the unlearning process incurs prohibitive overhead; and 2) updating all Low-Rank Adaptation (LoRA) modules indiscriminately leads to excessive resource consumption while potentially degrading model performance. To tackle these challenges, we propose a dual-level selective unlearning framework (DSU) for decentralized federated learning. At the agent level, DSU selects retained agents by matching historical LoRA update sketches, label sketches, and exposure to the withdrawn agent. Meanwhile, we derive a sensitivity score from the DFL training trajectory that measures each LoRA module’s accumulated influence from the target agent via the energy of its effective weight changes, and then update only the most sensitive modules while freezing the rest. Theoretical analysis relates selective unlearning to full-participation unlearning, and experiments show that DSU reduces the audited target-agent influence while preserving retained utility and reducing both participating agents and updated LoRA modules in the evaluated settings. The public reproducibility package is available at https://github.com/DGL-codes/LLM-DFL
Zhiqiang Xie, Yijing Lin, Zhipeng Gao et al.· IEEE Transactions on Cogniti...· 0 citations
Federated Learning is a distributed machine learning paradigm that trains a global model by aggregating local clients without sharing private data of each client. Federated Distillation (FD) builds upon this paradigm by leveraging knowledge distillation to exchange soft predictions on proxy data instead of model parameters, enabling more efficient communication and supporting heterogeneous model collaboration. However, FD models trained on In-Distribution data are hardly adapted to Out-of-Distribution (OOD) scenarios. In this paper, we propose a domain-aware proxy selection framework to better adopt proxy data for OOD problems. The experimental results show that the proposed models effectively address the challenges of distribution shifts under OOD with and without proxy data by achieving average 82.9\% and 80.6\% over existing works on standard benchmarks. The codes and data are released in https://anonymous.4open.science/r/DPS-FD-8596.
We present a comprehensive empirical study of federated learning (FL) for seven heterogeneous autonomous driving models and propose AdaFed, an adaptive selective aggregation strategy. We introduce a tier-based partial backbone sharing framework that groups models by shared architectural components (ResNet-34/50 backbones, bird’s-eye-view (BEV) encoders, and BEV-Lift modules) and evaluate six FL strategies: FedAvg, FedProx, SCAFFOLD, FedMD, FedDF, and AdaFed. Across all seven models, no FL strategy surpasses the per-model imitation-learning (IL) baseline; the practical question is therefore which strategy minimises degradation. AdaFed applies a tier-specific base strategy, blends the aggregated and local weights with an adaptive coefficient, and reverts any round-level update that worsens validation error. Experiments on CARLA 0.9.16 reveal that (1) partial backbone averaging stays closest to the IL baseline for ResNet-50 models (+0.05–0.13 m average displacement error (ADE)), (2) knowledge-distillation (KD) based approaches degrade performance, and (3) AdaFed yields the best FL result for the two models most exposed to negative transfer—VAD (2.05 m, −0.26 m vs. the best conventional FL strategy) and UniAD (1.52 m). These findings provide practical guidelines for heterogeneous FL deployment in autonomous driving.
Sunjun Hwang, Dohyun Hwang· International Conference on...· 0 citations
FedADB, a Class Anchor-Driven Dual-Branch FL framework, a dual-branch collaborative training mechanism designed for clients that achieves significant improvements in both accuracy and convergence speed.
Zhenyan Liu, Hua Zhang, Haoran Gao et al.· 0 citations
Personalized Federated Learning (pFL) has emerged as a promising paradigm, while existing approaches face 3 limitations: granularity mismatch, resource waste, and conflict aggregation. This paper presents a Personalized Federated Learning with Low-rank Pruning-based Adaptation (pFedLoPA) framework. Instead of balancing global and local trade-offs, pFedLoPA decouples the model into client-specific cores and globally shared complements. It integrates low-rank adaptation to constrain optimization to a compact subspace, gradient-based pruning to identify personalized parameters, and complementary aggregation to exchange only relevant updates. This enables clients to retain critical knowledge locally while efficiently integrating global knowledge. Extensive experiments on CIFAR-10/100 with multiple network architectures demonstrate that pFedLoPA outperforms state-of-the-art methods in test accuracy (up to 94.31% on CIFAR-10) while reducing communication costs by over 70%.
Luxi Cheng, Chuan Sun, Xiaohan Yuan et al.· Fall Joint Computer Conferen...· 0 citations