Ternary Compression for Communication-Efficient Federated Learning
- URL: http://arxiv.org/abs/2003.03564v2
- Date: Tue, 29 Mar 2022 08:50:30 GMT
- Title: Ternary Compression for Communication-Efficient Federated Learning
- Authors: Jinjin Xu, Wenli Du, Ran Cheng, Wangli He, Yaochu Jin
- Abstract summary: Federated learning provides a potential solution to privacy-preserving and secure machine learning.
We propose a ternary federated averaging protocol (T-FedAvg) to reduce the upstream and downstream communication of federated learning systems.
Our results show that the proposed T-FedAvg is effective in reducing communication costs and can even achieve slightly better performance on non-IID data.
- Score: 17.97683428517896
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Learning over massive data stored in different locations is essential in many
real-world applications. However, sharing data is full of challenges due to the
increasing demands of privacy and security with the growing use of smart mobile
devices and IoT devices. Federated learning provides a potential solution to
privacy-preserving and secure machine learning, by means of jointly training a
global model without uploading data distributed on multiple devices to a
central server. However, most existing work on federated learning adopts
machine learning models with full-precision weights, and almost all these
models contain a large number of redundant parameters that do not need to be
transmitted to the server, consuming an excessive amount of communication
costs. To address this issue, we propose a federated trained ternary
quantization (FTTQ) algorithm, which optimizes the quantized networks on the
clients through a self-learning quantization factor. Theoretical proofs of the
convergence of quantization factors, unbiasedness of FTTQ, as well as a reduced
weight divergence are given. On the basis of FTTQ, we propose a ternary
federated averaging protocol (T-FedAvg) to reduce the upstream and downstream
communication of federated learning systems. Empirical experiments are
conducted to train widely used deep learning models on publicly available
datasets, and our results demonstrate that the proposed T-FedAvg is effective
in reducing communication costs and can even achieve slightly better
performance on non-IID data in contrast to the canonical federated learning
algorithms.
Related papers
- FedSR: A Semi-Decentralized Federated Learning Algorithm for Non-IIDness in IoT System [2.040586739710704]
In the Industrial Internet of Things (IoT), a large amount of data will be generated every day.
Due to privacy and security issues, it is difficult to collect all these data together to train deep learning models.
In this paper, we combine centralized federated learning with decentralized federated learning to design a semi-decentralized cloud-edge-device hierarchical federated learning framework.
arXiv Detail & Related papers (2024-03-19T09:34:01Z) - Communication-Efficient Federated Learning through Adaptive Weight
Clustering and Server-Side Distillation [10.541541376305245]
Federated Learning (FL) is a promising technique for the collaborative training of deep neural networks across multiple devices.
FL is hindered by excessive communication costs due to repeated server-client communication during training.
We propose FedCompress, a novel approach that combines dynamic weight clustering and server-side knowledge distillation.
arXiv Detail & Related papers (2024-01-25T14:49:15Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - Adaptive Model Pruning and Personalization for Federated Learning over
Wireless Networks [72.59891661768177]
Federated learning (FL) enables distributed learning across edge devices while protecting data privacy.
We consider a FL framework with partial model pruning and personalization to overcome these challenges.
This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
arXiv Detail & Related papers (2023-09-04T21:10:45Z) - Federated Learning over Wireless IoT Networks with Optimized
Communication and Resources [98.18365881575805]
Federated learning (FL) as a paradigm of collaborative learning techniques has obtained increasing research attention.
It is of interest to investigate fast responding and accurate FL schemes over wireless systems.
We show that the proposed communication-efficient federated learning framework converges at a strong linear rate.
arXiv Detail & Related papers (2021-10-22T13:25:57Z) - Communication-Efficient Hierarchical Federated Learning for IoT
Heterogeneous Systems with Imbalanced Data [42.26599494940002]
Federated learning (FL) is a distributed learning methodology that allows multiple nodes to cooperatively train a deep learning model.
This paper studies the potential of hierarchical FL in IoT heterogeneous systems.
It proposes an optimized solution for user assignment and resource allocation on multiple edge nodes.
arXiv Detail & Related papers (2021-07-14T08:32:39Z) - CosSGD: Nonlinear Quantization for Communication-efficient Federated
Learning [62.65937719264881]
Federated learning facilitates learning across clients without transferring local data on these clients to a central server.
We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning.
Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
arXiv Detail & Related papers (2020-12-15T12:20:28Z) - WAFFLe: Weight Anonymized Factorization for Federated Learning [88.44939168851721]
In domains where data are sensitive or private, there is great value in methods that can learn in a distributed manner without the data ever leaving the local devices.
We propose Weight Anonymized Factorization for Federated Learning (WAFFLe), an approach that combines the Indian Buffet Process with a shared dictionary of weight factors for neural networks.
arXiv Detail & Related papers (2020-08-13T04:26:31Z) - Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G
Networks [84.2155885234293]
We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC.
To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
arXiv Detail & Related papers (2020-02-22T14:38:11Z) - Federated Learning with Cooperating Devices: A Consensus Approach for
Massive IoT Networks [8.456633924613456]
Federated learning (FL) is emerging as a new paradigm to train machine learning models in distributed systems.
The paper proposes a fully distributed (or server-less) learning approach: the proposed FL algorithms leverage the cooperation of devices that perform data operations inside the network.
The approach lays the groundwork for integration of FL within 5G and beyond networks characterized by decentralized connectivity and computing.
arXiv Detail & Related papers (2019-12-27T15:16:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.