Skip to content

FedFIbOS: Fisher Importance based Optimal Submodelling for Heterogeneous Federated Learning

Sep 2026 · 0 citations · 20 references
Computer Science

TL;DR

Experiments on CIFAR-10, CIFAR-100, and AGNews under pathological and Dirichlet non-IID settings show FedFIbOS achieves higher accuracy than the state of the art, with improvements becoming more pronounced under stronger heterogeneity.

Abstract

Heterogeneous federated learning requires clients with diverse computational capacities to collaboratively train a global model, where each client trains a capacity-constrained submodel. Existing methods select submodel parameters using heuristic importance measures---most prominently parameter magnitude---without theoretical justification for why these measures support convergence. We identify a fundamental gap: existing parameter selection criteria lack theoretical grounding in the convergence framework, partial client participation introduces additional estimation effects in the Fisher scores. We propose \textbf{FedFIbOS}: Fisher Importance-based Optimal Submodelling for heterogeneous federated learning, using Fisher Information in a principled criterion derived from minimizing submodel masking error. %We formally establish when magnitude selection is equivalent to Fisher selection fail under non-IID heterogeneous federated learning. We theoretically formulate submodel selection through a Fisher-weighted quadratic masking surrogate and show that the raw Fisher top-$k$ rule implemented by FedFIbOS solves this surrogate under a Fisher-dominant ranking condition. The resulting method retains the convergence structure of the underlying masked federated optimization bound. Fisher scores are efficiently estimated from empirical diagonal Fisher information using squared gradients, enabling stable and adaptive parameter selection without additional optimization overhead. Experiments on CIFAR-10, CIFAR-100, and AGNews under pathological and Dirichlet non-IID settings show FedFIbOS achieves ${\approx}10\%$ higher accuracy than the state of the art, with improvements becoming more pronounced under stronger heterogeneity.

View source

Similar papers

FedHV: Low-Overhead Hypervolume Weighting for Federated Multi-Objective Optimization

Task-wise federated multi-objective optimization (FedMOO) trains a shared model for competing prediction objectives under heterogeneous data, partial participation, and communication constraints. Existing methods commonly derive task weights from gradient or update geometry. This requires task-specific information or i...

A. Dehghanpour, Seyed Mohammad Azimi-Abarghouyi, Christopher G. Brinton · 0 citations
Conference Open access Sep 2026

Harmonizing Federated Heterogeneous Optimization via Adaptive Objective Rectification

Estimation error provably converges to zero as training progresses and HaFedHo surpasses state-of-the-art methods, including SCAFFOLD, MimeLite, and FedDyn, in both test accuracy and communication efficiency.

Jian-Rong Lu, Bang-Wei Li, Zhuo-Ya Gu et al. · 0 citations
Preprint Aug 2026

A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimization

A momentum-based variance-reduced algorithm for federated multiobjective optimization that incorporates a momentum-driven gradient estimator into the local updates to reduce the variance of stochastic updates, leading to an improved convergence rate.

Yong Zhao, Chunlin You, M. N. Dao et al. · 0 citations
Preprint Aug 2026

Beyond Parameter Space: NTK-Guided Personalized Aggregation for Robust Federated Learning

Local Inference Guided Aggregation for Heterogeneous Training Environments to Yield Enhancement Through Agreement and Regularization (LIGHTYEAR), a federated learning framework that performs update selection in function space using an NTK-based agreement score to characterize predictive behavior and determine a persona...

Mirko Konstantin, S. Zachow, Anirban Mukhopadhyay · 0 citations
Open access Aug 2026

A Meta-Learning-Based Aggregation Strategy for Heterogeneous Federated Learning Scenarios

It is demonstrated that prediction-space meta-learning constitutes a practical alternative for federated aggregation without requiring shared parameter initialization, and among the different forms of heterogeneity in federated learning, this work focuses on statistical and model heterogeneity.

Sergio Pérez-Picazo, Hiram Galeana-Zapién, Edwin Aldana-Bobadilla · 0 citations
Review Open access Sep 2026

Review on Aggregation Frameworks For High-Performance Federated Learning

This work establishes the research direction toward an adaptive weighted aggregation framework in which client contributions are dynamically adjusted using multiple performance-related indicators, expected to improve global accuracy, convergence stability, fairness, minority-class representation, and robustness to low-...

A. Wanare, Ir.Dr. Pankaj Agarka, Hemant A. Wani et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.