Release the Potential of Relative Feature Consistency for Data Heterogeneous Federated Learning

Abstract

Federated Learning is a promising paradigm to achieve privacy-preserving collaborative training. However, its performance degrades severely under data heterogeneity. A main obstacle is the divergent local representations induced by data heterogeneity, which will degrade the quality of the aggregated global representations. In this paper, we discover that although local representation spaces contain client-specific biases, their internal inter-class structure remains consistent across clients, which we identify as relative consistency. Through theoretical and empirical analyses, we further show that a proxy dataset can serve as a common anchor to translate class prototypes into a relative space, where client-specific shifts are suppressed and cross-client class centers become better aligned. Based on these findings, we propose a novel framework named Federated learning with Relative Representation Alignment (FedReRA) with Relative Knowledge Extraction (RKE) and Relative Ensemble Inference (REI) to extract and leverage the relative consistency to address data heterogeneity. Specifically, RKE debiases the prototypes by translating absolute prototypes into the aligned relative space via a flexible proxy dataset, while REI conducts collaborative inference with bias-reduced prediction in the relative space. Extensive experiments validate FedReRA's superiority over state-of-the-art methods, achieving a 3.83% average improvement under high heterogeneity and notably sustaining nearly 10% improvement on CIFAR-100 even when using random noise as the proxy data.