Convex Optimization Methods for Privacy-Preserving Federated Learning
TL;DR
This work provides a unified approach for aiding the design of state-of-the-art privacy-preserving distributed learning systems that are also utility-optimal and is an important step towards using such approaches in high-stakes domains like healthcare or finance.
Abstract
Federated Learning (FL) has emerged as a revolutionary paradigm in distributed machine learning, enabling multiple decentralized clients to collaboratively train models without sharing their local raw data. Despite its inherent privacy-centric design, FL remains vulnerable to sophisticated privacy attacks, such as gradient leakage and membership inference, which can reconstruct sensitive user data from communicated model updates. In order to reduce these vulnerabilities, we integrate privacy-preserving mechanisms most notably Differential Privacy (DP) and Cryptographic Protocols into the training procedure. These privacy constraints, however, come with utility loss and convergence slowdown thus highlighting a basic conflict between (differential) privacy on one side and high-order model accuracy and efficiency at another. In our paper, we carefully examine how to use convex optimization methods systematically in terms of performing this rich multi-dimensional trade-off. We center around the rigorous implementation of privacy-preserving FL couched as a bounded convex optimization task, studying how traditional and state-of-the-art optimization algorithms retain strong convergence rates even under durable privacy constraints. We benchmark the performance of these primary optimization frameworks, such as FedAvg, FedProx, and Accelerated Gradient Methods, when adopted on different privacy budgets. Theoretically, we analyze the impact of differential privacy on gradient variance in algorithms and experimentally validate how adaptive optimization (Specifically by AMSGrad) and proximal regularization can account for this noise-induced increase to enable faster convergence with a tight guarantee of differential privacy. To summarize, this work provides a unified approach for aiding the design of state-of-the-art privacy-preserving distributed learning systems that are also utility-optimal and is an important step towards using such approaches in high-stakes domains like healthcare or finance.