TL;DR
A federated learning framework based on self-excluding aggregation (FedSEA) is proposed by integrating dynamic sparsity-driven client grouping, self-excluding aggregation and cross-group knowledge distillation, which achieves superior performance on the Office-Home and CIFAR-10 datasets.
Knowledge Distillation (KD) is considered as an efficient way to replace the parameter averaging in federated learning, aiming to handle the clients with heterogeneous model architectures. Relying on the prepared distillation datasets across clients and the server, KD may encounter impractical difficulties in real-world implementations. Existing works explore the data-free KD in federated learning, which generates the distillation datasets on-site. However, the distillation datasets with global data distribution generated by these state-of-the-art schemes cannot be adapted to local non-IID data. In this article, we propose a new Personalized Data-Free Knowledge Distillation, namely PDKD, for federated learning under heterogeneous models and data. PDKD solves the problem of model drift caused by the inconsistent distribution of distillation datasets and the local data by generating personalized distillation datasets for each client while protecting client data privacy. In addition, we design a distillation dataset update scheme that maximizes the difference between teacher and client outputs on distillation datasets to accomplish deeper knowledge transfer. Furthermore, in order to accomplish the co-evolution of the teacher model and the clients’ model, PDKD incorporates a mutual distillation scheme. Numerous experiments show that PDKD significantly outperforms several state-of-the-art algorithms, with an 18% improvement in prediction accuracy and has a much lower communication cost than the compared algorithms.
Federated learning (FL) is an emerging distributed machine learning framework that enables collaborative learning among multiple parties while preserving data privacy. However, the complexity of environments and node heterogeneity in the real world result in uneven data distribution across nodes, leading to Non-IID (Non-Independent and Identically Distributed) characteristics in data distribution. Such data distribution significantly reduces the convergence and performance of the model, becoming one of the fundamental challenges in federated learning mechanisms. To address the above issue, this paper proposes a novel FL framework-FedGKD. For the Non-IID client data distribution problem, we employ client-side local data augmentation, where GAN models are deployed on each client to generate synthetic samples so that local data distribution imbalance can be effectively alleviated. To further overcome the limitations of client-side local data augmentation under Non-IID, FedGKD introduces server-side privacy-preserving data-free knowledge distillation, which can transfer the knowledge of selected clients to the server while ensuring privacy protection, further mitigating the impact of Non-IID on federated learning and solving the problem of model performance degradation caused by direct aggregation. Extensive experiments demonstrate that FedGKD significantly outperforms the baseline algorithms in terms of accuracy, while exhibiting excellent performance in other metrics.
Federated Learning is a distributed machine learning paradigm that trains a global model by aggregating local clients without sharing private data of each client. Federated Distillation (FD) builds upon this paradigm by leveraging knowledge distillation to exchange soft predictions on proxy data instead of model parameters, enabling more efficient communication and supporting heterogeneous model collaboration. However, FD models trained on In-Distribution data are hardly adapted to Out-of-Distribution (OOD) scenarios. In this paper, we propose a domain-aware proxy selection framework to better adopt proxy data for OOD problems. The experimental results show that the proposed models effectively address the challenges of distribution shifts under OOD with and without proxy data by achieving average 82.9\% and 80.6\% over existing works on standard benchmarks. The codes and data are released in https://anonymous.4open.science/r/DPS-FD-8596.
Overview: Federated Learning (FL) is a type of machine learning in which multiple clients use their own data to train a model. This does not require sending the raw data from each client to a central server. However, FL is currently very vulnerable to several types of attacks, including inference, model inversion, and membership inference. In this study, we have implemented a comprehensive experimental framework for analysing FL performance using standard FL aggregation protocols FedAvg, FedProx, and SCAFFOLD in conjunction with Differential Privacy (DP) mechanisms; specifically, the Gaussian noise mechanism with Rényi Differential Privacy (RDP) accountants. Our experiments all used K = 100 simulated clients, with heterogeneous data distribution across the entire federation (non-IID), and the respective clean datasets used were MNIST and CIFAR-10. For example, in our IID scenarios, our DP-FedAvg protocol achieved an accuracy of 84.9% on MNIST with ε = 1.0 and δ = 10^-5 (i.e., no privacy guarantees and 92.6% reuse of the same parameters). This corresponds to a 7.7-percentage-point trade-off in accuracy for provable privacy. In our non-IID scenario, we achieved an accuracy of 72.8% for the exact same privacy parameters and an LDA αᴸᴸᴸ = 0.5. We systematically analyse how various combinations of clipping norms on gradients, noise multipliers, and client participation rates converged. All results have been reported as-is, along with the associated standard deviation across each of the five independent sample runs that we performed using different random seed values. Finally, our framework will assist practitioners deploying privacy-friendly FL within either healthcare, banking, or Internet of Things environments.
Federated learning (FL) is a practical framework that can train models on distributed user data while guaranteeing data privacy; however, due to heterogeneity in which each user has a different data distribution, problems frequently arise where both global and personalization performance deteriorate simultaneously. This dissertation presents methodologies for building efficient personalized models by identifying which strategies are effective in the global training stage and by showing how to preserve global knowledge while securing user-specific performance during local adaptation. First, we show that as data heterogeneity increases, the collapse of feature vectors is a more fundamental bottleneck than classifier weights, and propose a method that directly mitigates the discrepancy in representation magnitude between local and global models. Second, we analyze that a training approach that strengthens local alignment can induce forgetting of global knowledge (e.g., categories not observed locally), and propose a method that achieves both local alignment and global knowledge preservation by combining feature distillation based on the global model's feature vectors. Third, in federated personalized reward model learning with preference heterogeneity, we empirically verify the conventional belief that"increasing the number of global models yields better initialization,"and we show that when sufficient local fine-tuning is allowed, a single global initialization can instead provide stronger personalization performance. This study redefines the role of global initialization under data and preference heterogeneity and provides practical training strategies that simultaneously satisfy global knowledge preservation and personalization.
Federated learning is a decentralised machine-learning approach in which several clients jointly build a shared model without moving their raw data to one location. Rising concerns around privacy, tightening regulation, and restrictions on how data may be owned or shared have made this approach increasingly attractive in practice. Although federated learning lowers privacy exposure relative to centralised training, deploying it in practice is complicated by clients whose data are unevenly distributed and non-identically distributed, by clients that participate inconsistently, and by training that can converge unpredictably. To obtain global models that train reliably and consistently even when client data are heterogeneous, this work puts forward a federated learning system built around privacy preservation. The design follows a client–server pattern in which a coordinating server aggregates updates from local models using weights that account for imbalance among participants. The behaviour of the resulting system is examined methodically across several data-distribution regimes — IID, mildly non-IID, and severely non-IID. The experiments show that the framework converges reliably and delivers predictive accuracy that holds up well, especially in the more difficult non-IID cases. Compared with conventional federated learning baselines, the approach shows greater robustness and steadier performance across successive training rounds. Because it is simple to implement, repeatable, and built with real deployment in mind, the architecture suits privacy-sensitive, decentralised use cases such as distributed intelligent systems, industrial monitoring, and healthcare analytics.