Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach

• URL: http://arxiv.org/abs/2302.04228v1
• Date: Wed, 8 Feb 2023 17:58:11 GMT
• Title: Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach
• Authors: Han Guo, Philip Greengard, Hongyi Wang, Andrew Gelman, Yoon Kim, Eric P. Xing
• Abstract summary: This paper extends the inference view and describes a variational inference formulation of federated learning. We apply FedEP on standard federated learning benchmarks and find that it outperforms strong baselines in terms of both convergence speed and accuracy.
• Score: 66.9033666087719
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The canonical formulation of federated learning treats it as a distributed optimization problem where the model parameters are optimized against a global loss function that decomposes across client loss functions. A recent alternative formulation instead treats federated learning as a distributed inference problem, where the goal is to infer a global posterior from partitioned client data (Al-Shedivat et al., 2021). This paper extends the inference view and describes a variational inference formulation of federated learning where the goal is to find a global variational posterior that well-approximates the true posterior. This naturally motivates an expectation propagation approach to federated learning (FedEP), where approximations to the global posterior are iteratively refined through probabilistic message-passing between the central server and the clients. We conduct an extensive empirical study across various algorithmic considerations and describe practical strategies for scaling up expectation propagation to the modern federated setting. We apply FedEP on standard federated learning benchmarks and find that it outperforms strong baselines in terms of both convergence speed and accuracy.

Related papers

• Aiding Global Convergence in Federated Learning via Local Perturbation and Mutual Similarity Information [6.767885381740953]
  Federated learning has emerged as a distributed optimization paradigm. We propose a novel modified framework wherein each client locally performs a perturbed gradient step. We show that our algorithm speeds convergence up to a margin of 30 global rounds compared with FedAvg.
  arXiv  Detail & Related papers  (2024-10-07T23:14:05Z)
• Adaptive Global-Local Representation Learning and Selection for Cross-Domain Facial Expression Recognition [54.334773598942775]
  Domain shift poses a significant challenge in Cross-Domain Facial Expression Recognition (CD-FER) We propose an Adaptive Global-Local Representation Learning and Selection framework.
  arXiv  Detail & Related papers  (2024-01-20T02:21:41Z)
• Aggregation Weighting of Federated Learning via Generalization Bound Estimation [65.8630966842025]
  Federated Learning (FL) typically aggregates client model parameters using a weighting approach determined by sample proportions. We replace the aforementioned weighting method with a new strategy that considers the generalization bounds of each local model.
  arXiv  Detail & Related papers  (2023-11-10T08:50:28Z)
• Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
  We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
  arXiv  Detail & Related papers  (2023-07-12T16:28:21Z)
• Bayesian Federated Neural Matching that Completes Full Information [2.6566593102111473]
  Federated learning is a machine learning paradigm where locally trained models are distilled into a global model. We propose a novel approach that overcomes this flaw by introducing a Kullback-Leibler divergence penalty at each iteration.
  arXiv  Detail & Related papers  (2022-11-15T09:47:56Z)
• DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixup [58.894901088797376]
  federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data. We propose a general framework to solve the above two challenges simultaneously. We provide comprehensive theoretical analysis including robustness analysis, convergence analysis, and generalization ability.
  arXiv  Detail & Related papers  (2022-04-16T08:08:29Z)
• Gradient Masked Averaging for Federated Learning [24.687254139644736]
  Federated learning allows a large number of clients with heterogeneous data to coordinate learning of a unified global model. Standard FL algorithms involve averaging of model parameters or gradient updates to approximate the global model at the server. We propose a gradient masked averaging approach for FL as an alternative to the standard averaging of client updates.
  arXiv  Detail & Related papers  (2022-01-28T08:42:43Z)
• Boosting Variational Inference With Locally Adaptive Step-Sizes [27.122745595473383]
  Boosting Variational Inference allows practitioners to obtain increasingly good posterior approximations by spending more compute. The main obstacle to widespread adoption of Boosting Variational Inference is the amount of resources necessary to improve over a strong Variational Inference baseline. We describe how the global curvature impacts time and memory consumption, address the problem with the notion of local curvature, and provide a novel approximate backtracking algorithm for estimating local curvature.
  arXiv  Detail & Related papers  (2021-05-19T16:41:33Z)
• Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms [21.11885845002748]
  We present an alternative perspective and formulate federated learning as a posterior inference problem. The goal is to infer a global posterior distribution by having client devices each infer the posterior of their local data. While exact inference is often intractable, this perspective provides a principled way to search for global optima in federated settings.
  arXiv  Detail & Related papers  (2020-10-11T15:55:45Z)
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization [93.78811018928583]
  This paper provides a framework to analyze the convergence of federated heterogeneous optimization algorithms. We propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.
  arXiv  Detail & Related papers  (2020-07-15T05:01:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.