Intrinisic Gradient Compression for Federated Learning
- URL: http://arxiv.org/abs/2112.02656v1
- Date: Sun, 5 Dec 2021 19:16:54 GMT
- Title: Intrinisic Gradient Compression for Federated Learning
- Authors: Luke Melas-Kyriazi, Franklyn Wang
- Abstract summary: Federated learning enables a large number of clients to jointly train a machine learning model on privately-held data.
One of the largest barriers to wider adoption of federated learning is the communication cost of sending model updates from and to the clients.
- Score: 3.9215337270154995
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated learning is a rapidly-growing area of research which enables a
large number of clients to jointly train a machine learning model on
privately-held data. One of the largest barriers to wider adoption of federated
learning is the communication cost of sending model updates from and to the
clients, which is accentuated by the fact that many of these devices are
bandwidth-constrained. In this paper, we aim to address this issue by
optimizing networks within a subspace of their full parameter space, an idea
known as intrinsic dimension in the machine learning theory community. We use a
correspondence between the notion of intrinsic dimension and gradient
compressibility to derive a family of low-bandwidth optimization algorithms,
which we call intrinsic gradient compression algorithms. Specifically, we
present three algorithms in this family with different levels of upload and
download bandwidth for use in various federated settings, along with
theoretical guarantees on their performance. Finally, in large-scale federated
learning experiments with models containing up to 100M parameters, we show that
our algorithms perform extremely well compared to current state-of-the-art
gradient compression methods.
Related papers
- Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner.
Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server.
This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
arXiv Detail & Related papers (2023-02-24T08:41:19Z) - Fed-LAMB: Layerwise and Dimensionwise Locally Adaptive Optimization
Algorithm [24.42828071396353]
In the emerging paradigm of federated learning (FL), large amount of clients, such as mobile devices, are used to train on their respective data.
Due to the low bandwidth, decentralized optimization methods need to shift the computation burden from those clients to those servers.
We present Fed-LAMB, a novel learning method based on a layerwise, deep neural networks.
arXiv Detail & Related papers (2021-10-01T16:54:31Z) - Comfetch: Federated Learning of Large Networks on Constrained Clients
via Sketching [28.990067638230254]
Federated learning (FL) is a popular paradigm for private and collaborative model training on the edge.
We propose a novel algorithm, Comdirectional, which allows clients to train large networks using representations of the global neural network.
arXiv Detail & Related papers (2021-09-17T04:48:42Z) - Sample-based and Feature-based Federated Learning via Mini-batch SSCA [18.11773963976481]
This paper investigates sample-based and feature-based federated optimization.
We show that the proposed algorithms can preserve data privacy through the model aggregation mechanism.
We also show that the proposed algorithms converge to Karush-Kuhn-Tucker points of the respective federated optimization problems.
arXiv Detail & Related papers (2021-04-13T08:23:46Z) - All at Once Network Quantization via Collaborative Knowledge Transfer [56.95849086170461]
We develop a novel collaborative knowledge transfer approach for efficiently training the all-at-once quantization network.
Specifically, we propose an adaptive selection strategy to choose a high-precision enquoteteacher for transferring knowledge to the low-precision student.
To effectively transfer knowledge, we develop a dynamic block swapping method by randomly replacing the blocks in the lower-precision student network with the corresponding blocks in the higher-precision teacher network.
arXiv Detail & Related papers (2021-03-02T03:09:03Z) - A Variational Information Bottleneck Based Method to Compress Sequential
Networks for Human Action Recognition [9.414818018857316]
We propose a method to effectively compress Recurrent Neural Networks (RNNs) used for Human Action Recognition (HAR)
We use a Variational Information Bottleneck (VIB) theory-based pruning approach to limit the information flow through the sequential cells of RNNs to a small subset.
We combine our pruning method with a specific group-lasso regularization technique that significantly improves compression.
It is shown that our method achieves over 70 times greater compression than the nearest competitor with comparable accuracy for the task of action recognition on UCF11.
arXiv Detail & Related papers (2020-10-03T12:41:51Z) - PowerGossip: Practical Low-Rank Communication Compression in
Decentralized Deep Learning [62.440827696638664]
We introduce a simple algorithm that directly compresses the model differences between neighboring workers.
Inspired by the PowerSGD for centralized deep learning, this algorithm uses power steps to maximize the information transferred per bit.
arXiv Detail & Related papers (2020-08-04T09:14:52Z) - ALF: Autoencoder-based Low-rank Filter-sharing for Efficient
Convolutional Neural Networks [63.91384986073851]
We propose the autoencoder-based low-rank filter-sharing technique technique (ALF)
ALF shows a reduction of 70% in network parameters, 61% in operations and 41% in execution time, with minimal loss in accuracy.
arXiv Detail & Related papers (2020-07-27T09:01:22Z) - Communication-Efficient Distributed Stochastic AUC Maximization with
Deep Neural Networks [50.42141893913188]
We study a distributed variable for large-scale AUC for a neural network as with a deep neural network.
Our model requires a much less number of communication rounds and still a number of communication rounds in theory.
Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
arXiv Detail & Related papers (2020-05-05T18:08:23Z) - Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of
Partitioned Edge Learning [73.82875010696849]
Machine learning algorithms are deployed at the network edge for training artificial intelligence (AI) models.
This paper focuses on the novel joint design of parameter (computation load) allocation and bandwidth allocation.
arXiv Detail & Related papers (2020-03-10T05:52:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.