Stochastic Approximation Approach to Federated Machine Learning
- URL: http://arxiv.org/abs/2402.12945v1
- Date: Tue, 20 Feb 2024 12:00:25 GMT
- Title: Stochastic Approximation Approach to Federated Machine Learning
- Authors: Srihari P V and Bharath Bhikkaji
- Abstract summary: This paper examines Federated learning (FL) in a Approximation (SA) framework.
FL is a collaborative way to train neural network models across various participants or clients.
It is observed that the proposed algorithm is robust and gives more reliable estimates of the weights.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper examines Federated learning (FL) in a Stochastic Approximation
(SA) framework. FL is a collaborative way to train neural network models across
various participants or clients without centralizing their data. Each client
will train a model on their respective data and send the weights across to a
the server periodically for aggregation. The server aggregates these weights
which are then used by the clients to re-initialize their neural network and
continue the training. SA is an iterative algorithm that uses approximate
sample gradients and tapering step size to locate a minimizer of a cost
function. In this paper the clients use a stochastic approximation iterate to
update the weights of its neural network. It is shown that the aggregated
weights track an autonomous ODE. Numerical simulations are performed and the
results are compared with standard algorithms like FedAvg and FedProx. It is
observed that the proposed algorithm is robust and gives more reliable
estimates of the weights, in particular when the clients data are not
identically distributed.
Related papers
- Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning [51.560590617691005]
We investigate whether it is possible to squeeze more juice" out of each cohort than what is possible in a single communication round.
Our approach leads to up to 74% reduction in the total communication cost needed to train a FL model in the cross-device setting.
arXiv Detail & Related papers (2024-06-03T08:48:49Z) - Federated Learning on Non-iid Data via Local and Global Distillation [25.397058380098816]
We propose FedND: federated learning with noise distillation.
In the client, we propose a self-distillation method to train the local model.
In the server, we generate noisy samples for each client and use them to distill other clients.
Experimental results show that the algorithm achieves the best performance and is more communication-efficient than state-of-the-art methods.
arXiv Detail & Related papers (2023-06-26T06:14:01Z) - Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device)
In FL, each data holder trains a model locally and releases it to a central server for aggregation.
In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation).
In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
arXiv Detail & Related papers (2022-11-20T10:49:22Z) - Sparse Random Networks for Communication-Efficient Federated Learning [23.614934319624826]
One main challenge in federated learning is the large communication cost of exchanging weight updates from clients to the server at each round.
We propose a radically different approach that does not update the weights at all.
Instead, our method freezes the weights at their initial emphrandom values and learns how to sparsify the random network for the best performance.
arXiv Detail & Related papers (2022-09-30T09:11:09Z) - Federated Stochastic Gradient Descent Begets Self-Induced Momentum [151.4322255230084]
Federated learning (FL) is an emerging machine learning method that can be applied in mobile edge systems.
We show that running to the gradient descent (SGD) in such a setting can be viewed as adding a momentum-like term to the global aggregation process.
arXiv Detail & Related papers (2022-02-17T02:01:37Z) - An Expectation-Maximization Perspective on Federated Learning [75.67515842938299]
Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device.
In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters.
We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
arXiv Detail & Related papers (2021-11-19T12:58:59Z) - Training Sparse Neural Networks using Compressed Sensing [13.84396596420605]
We develop and test a novel method based on compressed sensing which combines the pruning and training into a single step.
Specifically, we utilize an adaptively weighted $ell1$ penalty on the weights during training, which we combine with a generalization of the regularized dual averaging (RDA) algorithm in order to train sparse neural networks.
arXiv Detail & Related papers (2020-08-21T19:35:54Z) - Superiority of Simplicity: A Lightweight Model for Network Device
Workload Prediction [58.98112070128482]
We propose a lightweight solution for series prediction based on historic observations.
It consists of a heterogeneous ensemble method composed of two models - a neural network and a mean predictor.
It achieves an overall $R2$ score of 0.10 on the available FedCSIS 2020 challenge dataset.
arXiv Detail & Related papers (2020-07-07T15:44:16Z) - Towards Federated Learning With Byzantine-Robust Client Weighting [2.294014185517203]
We propose a practical weight-truncation-based preprocessing method for Federated Learning.
We show that our method can strike a good balance between model quality and Byzantine robustness.
arXiv Detail & Related papers (2020-04-10T10:59:16Z) - Model Fusion via Optimal Transport [64.13185244219353]
We present a layer-wise model fusion algorithm for neural networks.
We show that this can successfully yield "one-shot" knowledge transfer between neural networks trained on heterogeneous non-i.i.d. data.
arXiv Detail & Related papers (2019-10-12T22:07:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.