Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information

• URL: http://arxiv.org/abs/2012.03178v1
• Date: Sun, 6 Dec 2020 03:54:58 GMT
• Title: Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information
• Authors: Peng Xiao, Samuel Cheng
• Abstract summary: In federated learning, models trained on local clients are distilled into a global model. We propose a new method which extends the Probabilistic Federated Neural Matching. Our new method outperforms popular state-of-the-art federated learning methods in both single communication round and additional communication rounds situation.
• Score: 4.067903810030317
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In federated learning, models trained on local clients are distilled into a global model. Due to the permutation invariance arises in neural networks, it is necessary to match the hidden neurons first when executing federated learning with neural networks. Through the Bayesian nonparametric framework, Probabilistic Federated Neural Matching (PFNM) matches and fuses local neural networks so as to adapt to varying global model size and the heterogeneity of the data. In this paper, we propose a new method which extends the PFNM with a Kullback-Leibler (KL) divergence over neural components product, in order to make inference exploiting posterior information in both local and global levels. We also show theoretically that The additional part can be seamlessly concatenated into the match-and-fuse progress. Through a series of simulations, it indicates that our new method outperforms popular state-of-the-art federated learning methods in both single communication round and additional communication rounds situation.

Related papers

• FedPNN: One-shot Federated Classification via Evolving Clustering Method and Probabilistic Neural Network hybrid [4.241208172557663]
  We propose a two-stage federated learning approach toward the objective of privacy protection. In the first stage, the synthetic dataset is generated by employing two different distributions as noise. In the second stage, the Federated Probabilistic Neural Network (FedPNN) is developed and employed for building globally shared classification model.
  arXiv  Detail & Related papers  (2023-04-09T03:23:37Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Bayesian Federated Neural Matching that Completes Full Information [2.6566593102111473]
  Federated learning is a machine learning paradigm where locally trained models are distilled into a global model. We propose a novel approach that overcomes this flaw by introducing a Kullback-Leibler divergence penalty at each iteration.
  arXiv  Detail & Related papers  (2022-11-15T09:47:56Z)
• Investigating Neuron Disturbing in Fusing Heterogeneous Neural Networks [6.389882065284252]
  In this paper, we reveal the phenomenon of neuron disturbing, where neurons from heterogeneous local models interfere with each other mutually. We propose an experimental method that excludes neuron disturbing and fuses neural networks via adaptively selecting a local model, called AMS, to execute the prediction.
  arXiv  Detail & Related papers  (2022-10-24T06:47:48Z)
• Combined Learning of Neural Network Weights for Privacy in Collaborative Tasks [1.1172382217477126]
  CoLN, Combined Learning of Neural network weights, is a novel method to securely combine Machine Learning models over sensitive data with no sharing of data. CoLN can contribute to secure collaborative research, as required in the medical area, where privacy issues preclude data sharing.
  arXiv  Detail & Related papers  (2022-04-30T22:40:56Z)
• On Feature Learning in Neural Networks with Global Convergence Guarantees [49.870593940818715]
  We study the optimization of wide neural networks (NNs) via gradient flow (GF) We show that when the input dimension is no less than the size of the training set, the training loss converges to zero at a linear rate under GF. We also show empirically that, unlike in the Neural Tangent Kernel (NTK) regime, our multi-layer model exhibits feature learning and can achieve better generalization performance than its NTK counterpart.
  arXiv  Detail & Related papers  (2022-04-22T15:56:43Z)
• Clustered Federated Learning via Generalized Total Variation Minimization [83.26141667853057]
  We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure. Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV) Our main algorithmic contribution is a fully decentralized federated learning algorithm.
  arXiv  Detail & Related papers  (2021-05-26T18:07:19Z)
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks [94.69202357137452]
  We propose a novel model-parallel learning method, called local critic training. We show that the proposed approach successfully decouples the update process of the layer groups for both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) We also show that trained networks by the proposed method can be used for structural optimization.
  arXiv  Detail & Related papers  (2021-02-03T09:30:45Z)
• Neural Networks Enhancement with Logical Knowledge [83.9217787335878]
  We propose an extension of KENN for relational data. The results show that KENN is capable of increasing the performances of the underlying neural network even in the presence relational data.
  arXiv  Detail & Related papers  (2020-09-13T21:12:20Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• Federated Learning with Matched Averaging [43.509797844077426]
  Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device. We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures.
  arXiv  Detail & Related papers  (2020-02-15T20:09:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.