Dynamic Gradient Aggregation for Federated Domain Adaptation

• URL: http://arxiv.org/abs/2106.07578v1
• Date: Mon, 14 Jun 2021 16:34:28 GMT
• Title: Dynamic Gradient Aggregation for Federated Domain Adaptation
• Authors: Dimitrios Dimitriadis, Kenichi Kumatani, Robert Gmyr, Yashesh Gaur and Sefik Emre Eskimez
• Abstract summary: We introduce a new learning algorithm for Federated Learning (FL) The proposed scheme is based on a weighted gradient aggregation using two-step optimization to offer a flexible training pipeline. We investigate the effect of our FL algorithm in supervised and unsupervised Speech Recognition (SR) scenarios.
• Score: 31.264050568762592
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, a new learning algorithm for Federated Learning (FL) is introduced. The proposed scheme is based on a weighted gradient aggregation using two-step optimization to offer a flexible training pipeline. Herein, two different flavors of the aggregation method are presented, leading to an order of magnitude improvement in convergence speed compared to other distributed or FL training algorithms like BMUF and FedAvg. Further, the aggregation algorithm acts as a regularizer of the gradient quality. We investigate the effect of our FL algorithm in supervised and unsupervised Speech Recognition (SR) scenarios. The experimental validation is performed based on three tasks: first, the LibriSpeech task showing a speed-up of 7x and 6% word error rate reduction (WERR) compared to the baseline results. The second task is based on session adaptation providing 20% WERR over a powerful LAS model. Finally, our unsupervised pipeline is applied to the conversational SR task. The proposed FL system outperforms the baseline systems in both convergence speed and overall model performance.

Related papers

• Efficient Diffusion as Low Light Enhancer [63.789138528062225]
  Reflectance-Aware Trajectory Refinement (RATR) is a simple yet effective module to refine the teacher trajectory using the reflectance component of images. textbfReflectance-aware textbfDiffusion with textbfDistilled textbfTrajectory (textbfReDDiT) is an efficient and flexible distillation framework tailored for Low-Light Image Enhancement (LLIE)
  arXiv  Detail & Related papers  (2024-10-16T08:07:18Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Optimal feature rescaling in machine learning based on neural networks [0.0]
  An optimal rescaling of input features (OFR) is carried out by a Genetic Algorithm (GA) The OFR reshapes the input space improving the conditioning of the gradient-based algorithm used for the training. The approach has been tested on a FFNN modeling the outcome of a real industrial process.
  arXiv  Detail & Related papers  (2024-02-13T21:57:31Z)
• Stochastic Unrolled Federated Learning [85.6993263983062]
  We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning. Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
  arXiv  Detail & Related papers  (2023-05-24T17:26:22Z)
• FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averaging [104.41634756395545]
  Federated learning (FL) is an emerging learning paradigm to tackle massively distributed data. We propose textbfFedDA, a novel framework for local adaptive gradient methods. We show that textbfFedDA-MVR is the first adaptive FL algorithm that achieves this rate.
  arXiv  Detail & Related papers  (2023-02-13T05:10:30Z)
• Faster Adaptive Federated Learning [84.38913517122619]
  Federated learning has attracted increasing attention with the emergence of distributed data. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
  arXiv  Detail & Related papers  (2022-12-02T05:07:50Z)
• Federated Transfer Learning with Dynamic Gradient Aggregation [27.42998421786922]
  This paper introduces a Federated Learning (FL) simulation platform for Acoustic Model training. The proposed FL platform can support different tasks based on the adopted modular design. It is shown to outperform the golden standard of distributed training in both convergence speed and overall model performance.
  arXiv  Detail & Related papers  (2020-08-06T04:29:01Z)
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks [50.42141893913188]
  We study a distributed variable for large-scale AUC for a neural network as with a deep neural network. Our model requires a much less number of communication rounds and still a number of communication rounds in theory. Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
  arXiv  Detail & Related papers  (2020-05-05T18:08:23Z)
• Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning [8.366415386275557]
  Solution involves a reformation of the objective function for optimization in neural network models. We introduce a decentralized weighted aggregating scheme based on the performance of local workers. To validate the new method, we benchmark our schemes against several popular algorithms.
  arXiv  Detail & Related papers  (2020-04-07T23:38:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.