Federated Generalized Bayesian Learning via Distributed Stein
Variational Gradient Descent
- URL: http://arxiv.org/abs/2009.06419v6
- Date: Tue, 30 Mar 2021 13:14:24 GMT
- Title: Federated Generalized Bayesian Learning via Distributed Stein
Variational Gradient Descent
- Authors: Rahif Kassab and Osvaldo Simeone
- Abstract summary: This paper introduces Distributed Stein Variational Gradient Descent (DSVGD), a non-parametric generalized Bayesian inference framework for federated learning.
By varying the number of particles, DSVGD enables a flexible trade-off between per-iteration communication load and number of communication rounds.
- Score: 38.41707037232561
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper introduces Distributed Stein Variational Gradient Descent (DSVGD),
a non-parametric generalized Bayesian inference framework for federated
learning. DSVGD maintains a number of non-random and interacting particles at a
central server to represent the current iterate of the model global posterior.
The particles are iteratively downloaded and updated by one of the agents with
the end goal of minimizing the global free energy. By varying the number of
particles, DSVGD enables a flexible trade-off between per-iteration
communication load and number of communication rounds. DSVGD is shown to
compare favorably to benchmark frequentist and Bayesian federated learning
strategies, also scheduling a single device per iteration, in terms of accuracy
and scalability with respect to the number of agents, while also providing
well-calibrated, and hence trustworthy, predictions.
Related papers
- Accelerating Convergence of Stein Variational Gradient Descent via Deep
Unfolding [5.584060970507506]
Stein variational gradient descent (SVGD) is a prominent particle-based variational inference method used for sampling a target distribution.
In this paper, we propose novel trainable algorithms that incorporate a deep-learning technique called deep unfolding,into SVGD.
arXiv Detail & Related papers (2024-02-23T06:24:57Z) - Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized
Language Model Finetuning Using Shared Randomness [86.61582747039053]
Language model training in distributed settings is limited by the communication cost of exchanges.
We extend recent work using shared randomness to perform distributed fine-tuning with low bandwidth.
arXiv Detail & Related papers (2023-06-16T17:59:51Z) - Augmented Message Passing Stein Variational Gradient Descent [3.5788754401889014]
We study the isotropy property of finite particles during the convergence process.
All particles tend to cluster around the particle center within a certain range.
Our algorithm achieves satisfactory accuracy and overcomes the variance collapse problem in various benchmark problems.
arXiv Detail & Related papers (2023-05-18T01:13:04Z) - Client Selection for Federated Bayesian Learning [6.055038050790775]
We propose two selection schemes for DSVGD based on Kernelized Stein Discrepancy (KSD) and Hilbert Inner Product (HIP)
We evaluate and compare our schemes with conventional schemes in terms of model accuracy, convergence speed, and stability using various learning tasks and datasets.
arXiv Detail & Related papers (2022-12-11T12:37:31Z) - From Points to Functions: Infinite-dimensional Representations in
Diffusion Models [23.916417852496608]
Diffusion-based generative models learn to iteratively transfer unstructured noise to a complex target distribution.
We show that a combination of information content from different time steps gives a strictly better representation for the downstream task.
arXiv Detail & Related papers (2022-10-25T05:30:53Z) - Scaling Structured Inference with Randomization [64.18063627155128]
We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states.
Our method is widely applicable to classical DP-based inference.
It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
arXiv Detail & Related papers (2021-12-07T11:26:41Z) - Forget-SVGD: Particle-Based Bayesian Federated Unlearning [32.638916321653554]
Forget-Stein Variational Gradient Descent (Forget-SVGD) builds on SVGD.
The proposed method is validated via performance comparisons with non-parametric schemes that train from scratch by excluding data to be forgotten.
arXiv Detail & Related papers (2021-11-23T18:15:50Z) - DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic
Convolution [136.7261709896713]
We propose a data-driven approach that generates the appropriate convolution kernels to apply in response to the nature of the instances.
The proposed method achieves promising results on both ScanetNetV2 and S3DIS.
It also improves inference speed by more than 25% over the current state-of-the-art.
arXiv Detail & Related papers (2020-11-26T14:56:57Z) - Kernel Stein Generative Modeling [68.03537693810972]
Gradient Langevin Dynamics (SGLD) demonstrates impressive results with energy-based models on high-dimensional and complex data distributions.
Stein Variational Gradient Descent (SVGD) is a deterministic sampling algorithm that iteratively transports a set of particles to approximate a given distribution.
We propose noise conditional kernel SVGD (NCK-SVGD), that works in tandem with the recently introduced Noise Conditional Score Network estimator.
arXiv Detail & Related papers (2020-07-06T21:26:04Z) - Stein Variational Inference for Discrete Distributions [70.19352762933259]
We propose a simple yet general framework that transforms discrete distributions to equivalent piecewise continuous distributions.
Our method outperforms traditional algorithms such as Gibbs sampling and discontinuous Hamiltonian Monte Carlo.
We demonstrate that our method provides a promising tool for learning ensembles of binarized neural network (BNN)
In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions.
arXiv Detail & Related papers (2020-03-01T22:45:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.