Enabling On-Device Training of Speech Recognition Models with Federated Dropout

• URL: http://arxiv.org/abs/2110.03634v1
• Date: Thu, 7 Oct 2021 17:22:40 GMT
• Title: Enabling On-Device Training of Speech Recognition Models with Federated Dropout
• Authors: Dhruv Guliani and Lillian Zhou and Changwan Ryu and Tien-Ju Yang and Harry Zhang and Yonghui Xiao and Francoise Beaufays and Giovanni Motta
• Abstract summary: Federated learning can be used to train machine learning models on the edge on local data that never leave devices. We propose using federated dropout to reduce the size of client models while training a full-size model server-side.
• Score: 4.165917555996752
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning can be used to train machine learning models on the edge on local data that never leave devices, providing privacy by default. This presents a challenge pertaining to the communication and computation costs associated with clients' devices. These costs are strongly correlated with the size of the model being trained, and are significant for state-of-the-art automatic speech recognition models. We propose using federated dropout to reduce the size of client models while training a full-size model server-side. We provide empirical evidence of the effectiveness of federated dropout, and propose a novel approach to vary the dropout rate applied at each layer. Furthermore, we find that federated dropout enables a set of smaller sub-models within the larger model to independently have low word error rates, making it easier to dynamically adjust the size of the model deployed for inference.

Related papers

• Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
  Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources. The lack of model privacy protection in FL becomes an unneglectable challenge. We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
  arXiv  Detail & Related papers  (2023-11-12T11:01:10Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Federated Pruning: Improving Neural Network Efficiency with Federated Learning [24.36174705715827]
  We propose Federated Pruning to train a reduced model under the federated setting. We explore different pruning schemes and provide empirical evidence of the effectiveness of our methods.
  arXiv  Detail & Related papers  (2022-09-14T00:48:37Z)
• Towards Sparsified Federated Neuroimaging Models via Weight Pruning [13.0319103091844]
  FedSparsify performs model pruning during federated training. We show that models can be pruned up to 95% sparsity without affecting performance. One surprising benefit of model pruning is improved model privacy.
  arXiv  Detail & Related papers  (2022-08-24T17:05:47Z)
• Federated Learning of Neural ODE Models with Different Iteration Counts [0.9444784653236158]
  Federated learning is a distributed machine learning approach in which clients train models locally with their own data and upload them to a server so that their trained results are shared between them without uploading raw data to the server. In this paper, we utilize Neural ODE based models for federated learning. We show that our approach can reduce communication size by up to 92.4% compared with a baseline ResNet model using CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2022-08-19T17:57:32Z)
• Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices [14.604785223644718]
  Federated Learning (FL) is a technique to train models using data distributed across devices. Differential Privacy (DP) provides a formal privacy guarantee for sensitive data. We propose Partial Embedding Updates (PEU) to decrease noise by decreasing payload size.
  arXiv  Detail & Related papers  (2022-07-18T23:53:17Z)
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices [79.16481453598266]
  We propose InclusiveFL, a client-inclusive federated learning method to handle this problem. The core idea of InclusiveFL is to assign models of different sizes to clients with different computing capabilities. We also propose an effective method to share the knowledge among multiple local models with different sizes.
  arXiv  Detail & Related papers  (2022-02-16T13:03:27Z)
• FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients [41.623518032533035]
  In split learning, only a small part of the model is stored and trained on clients while the remaining large part of the model only stays at the servers. This paper addresses this issue by compressing the additional communication using a novel clustering scheme accompanied by a gradient correction method.
  arXiv  Detail & Related papers  (2022-01-28T00:09:53Z)
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation [56.886414139084216]
  Federated learning is widely used to learn intelligent models from decentralized data. In federated learning, clients need to communicate their local model updates in each iteration of model learning. We propose a communication efficient federated learning method based on knowledge distillation.
  arXiv  Detail & Related papers  (2021-08-30T15:39:54Z)
• UVeQFed: Universal Vector Quantization for Federated Learning [179.06583469293386]
  Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their possibly private labeled data. In FL, each user trains its copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion.
  arXiv  Detail & Related papers  (2020-06-05T07:10:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.