Efficient and Private Federated Learning with Partially Trainable Networks

• URL: http://arxiv.org/abs/2110.03450v1
• Date: Wed, 6 Oct 2021 04:28:33 GMT
• Title: Efficient and Private Federated Learning with Partially Trainable Networks
• Authors: Hakim Sidahmed, Zheng Xu, Ankush Garg, Yuan Cao, Mingqing Chen
• Abstract summary: We propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process. We empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs. Our approach also enables faster training, with a smaller memory footprint, and better utility for strong differential privacy guarantees.
• Score: 8.813191488656527
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning is used for decentralized training of machine learning models on a large number (millions) of edge mobile devices. It is challenging because mobile devices often have limited communication bandwidth and local computation resources. Therefore, improving the efficiency of federated learning is critical for scalability and usability. In this paper, we propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process, to reduce the communication cost with little implications on model performance. Through extensive experiments, we empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs, with up to $46\times$ reduction in communication cost, at a small accuracy cost. Our approach also enables faster training, with a smaller memory footprint, and better utility for strong differential privacy guarantees. The proposed FedPT method can be particularly interesting for pushing the limitations of overparameterization in on-device learning.

Related papers

• Edge-assisted U-Shaped Split Federated Learning with Privacy-preserving for Internet of Things [4.68267059122563]
  We present an innovative Edge-assisted U-Shaped Split Federated Learning (EUSFL) framework, which harnesses the high-performance capabilities of edge servers. In this framework, we leverage Federated Learning (FL) to enable data holders to collaboratively train models without sharing their data. We also propose a novel noise mechanism called LabelDP to ensure that data features and labels can securely resist reconstruction attacks.
  arXiv  Detail & Related papers  (2023-11-08T05:14:41Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Towards Federated Learning Under Resource Constraints via Layer-wise Training and Depth Dropout [33.308067180286045]
  Federated learning can be difficult to scale to large models when clients have limited resources. We introduce Federated Layer-wise Learning to simultaneously reduce per-client memory, computation, and communication costs. We also introduce Federated Depth Dropout, a complementary technique that randomly drops frozen layers during training, to further reduce resource usage.
  arXiv  Detail & Related papers  (2023-09-11T03:17:45Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• Federated Pruning: Improving Neural Network Efficiency with Federated Learning [24.36174705715827]
  We propose Federated Pruning to train a reduced model under the federated setting. We explore different pruning schemes and provide empirical evidence of the effectiveness of our methods.
  arXiv  Detail & Related papers  (2022-09-14T00:48:37Z)
• Online Data Selection for Federated Learning with Limited Storage [53.46789303416799]
  Federated Learning (FL) has been proposed to achieve distributed machine learning among networked devices. The impact of on-device storage on the performance of FL is still not explored. In this work, we take the first step to consider the online data selection for FL with limited on-device storage.
  arXiv  Detail & Related papers  (2022-09-01T03:27:33Z)
• Federated Learning over Wireless IoT Networks with Optimized Communication and Resources [98.18365881575805]
  Federated learning (FL) as a paradigm of collaborative learning techniques has obtained increasing research attention. It is of interest to investigate fast responding and accurate FL schemes over wireless systems. We show that the proposed communication-efficient federated learning framework converges at a strong linear rate.
  arXiv  Detail & Related papers  (2021-10-22T13:25:57Z)
• Partial Variable Training for Efficient On-Device Federated Learning [3.884530687475797]
  We propose a novel method, called Partial Variable Training (PVT), that only trains a small subset of variables on edge devices to reduce memory usage and communication cost. According to experiments on two state-of-the-art neural networks for speech recognition and two different datasets, PVT can reduce memory usage by up to 1.9$times$ and communication cost by up to 593$times$ while attaining comparable accuracy when compared with full network training.
  arXiv  Detail & Related papers  (2021-10-11T20:57:06Z)
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training [65.68511423300812]
  We propose ProgFed, a progressive training framework for efficient and effective federated learning. ProgFed inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. Our results show that ProgFed converges at the same rate as standard training on full models.
  arXiv  Detail & Related papers  (2021-10-11T14:45:00Z)
• CosSGD: Nonlinear Quantization for Communication-efficient Federated Learning [62.65937719264881]
  Federated learning facilitates learning across clients without transferring local data on these clients to a central server. We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning. Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
  arXiv  Detail & Related papers  (2020-12-15T12:20:28Z)
• Ternary Compression for Communication-Efficient Federated Learning [17.97683428517896]
  Federated learning provides a potential solution to privacy-preserving and secure machine learning. We propose a ternary federated averaging protocol (T-FedAvg) to reduce the upstream and downstream communication of federated learning systems. Our results show that the proposed T-FedAvg is effective in reducing communication costs and can even achieve slightly better performance on non-IID data.
  arXiv  Detail & Related papers  (2020-03-07T11:55:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.