Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data

• URL: http://arxiv.org/abs/2009.06557v2
• Date: Tue, 22 Dec 2020 01:29:59 GMT
• Title: Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
• Authors: Qianqian Tong, Guannan Liang and Jinbo Bi
• Abstract summary: Federated learning allows devices to collaboratively learn a model without data sharing. We propose Federated AGMs, which employ both the firstorder and second-ordercalibratea. We compare schemes of calibration for federated learning, including standard Adam byepsilon.
• Score: 18.678289386084113
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning allows loads of edge computing devices to collaboratively learn a global model without data sharing. The analysis with partial device participation under non-IID and unbalanced data reflects more reality. In this work, we propose federated learning versions of adaptive gradient methods - Federated AGMs - which employ both the first-order and second-order momenta, to alleviate generalization performance deterioration caused by dissimilarity of data population among devices. To further improve the test performance, we compare several schemes of calibration for the adaptive learning rate, including the standard Adam calibrated by $\epsilon$, $p$-Adam, and one calibrated by an activation function. Our analysis provides the first set of theoretical results that the proposed (calibrated) Federated AGMs converge to a first-order stationary point under non-IID and unbalanced data settings for nonconvex optimization. We perform extensive experiments to compare these federated learning methods with the state-of-the-art FedAvg, FedMomentum and SCAFFOLD and to assess the different calibration schemes and the advantages of AGMs over the current federated learning methods.

Related papers

• Parameter Tracking in Federated Learning with Adaptive Optimization [14.111863825607001]
  In Federated Learning (FL), model training performance is strongly impacted by data heterogeneity across clients. Gradient Tracking (GT) has recently emerged as a solution which mitigates this issue by introducing correction terms to local model updates. To date, GT has only been considered under Gradient (SGD)-based model Descent training, while modern FL frameworks increasingly employ adaptives for improved convergence.
  arXiv  Detail & Related papers  (2025-02-04T21:21:30Z)
• Non-Convex Optimization in Federated Learning via Variance Reduction and Adaptive Learning [13.83895180419626]
  This paper proposes a novel algorithm that leverages momentum-based variance reduction with adaptive learning to address non-epsilon settings across heterogeneous data. We aim to overcome challenges related to variance, hinders efficiency, and the slow convergence from learning rate adjustments with heterogeneous data.
  arXiv  Detail & Related papers  (2024-12-16T11:02:38Z)
• Aiding Global Convergence in Federated Learning via Local Perturbation and Mutual Similarity Information [6.767885381740953]
  Federated learning has emerged as a distributed optimization paradigm. We propose a novel modified framework wherein each client locally performs a perturbed gradient step. We show that our algorithm speeds convergence up to a margin of 30 global rounds compared with FedAvg.
  arXiv  Detail & Related papers  (2024-10-07T23:14:05Z)
• Filling the Missing: Exploring Generative AI for Enhanced Federated Learning over Heterogeneous Mobile Edge Devices [72.61177465035031]
  We propose a generative AI-empowered federated learning to address these challenges by leveraging the idea of FIlling the MIssing (FIMI) portion of local data. Experiment results demonstrate that FIMI can save up to 50% of the device-side energy to achieve the target global test accuracy.
  arXiv  Detail & Related papers  (2023-10-21T12:07:04Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Federated Compositional Deep AUC Maximization [58.25078060952361]
  We develop a novel federated learning method for imbalanced data by directly optimizing the area under curve (AUC) score. To the best of our knowledge, this is the first work to achieve such favorable theoretical results.
  arXiv  Detail & Related papers  (2023-04-20T05:49:41Z)
• Faster Adaptive Federated Learning [84.38913517122619]
  Federated learning has attracted increasing attention with the emergence of distributed data. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
  arXiv  Detail & Related papers  (2022-12-02T05:07:50Z)
• DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixup [58.894901088797376]
  federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data. We propose a general framework to solve the above two challenges simultaneously. We provide comprehensive theoretical analysis including robustness analysis, convergence analysis, and generalization ability.
  arXiv  Detail & Related papers  (2022-04-16T08:08:29Z)
• Heterogeneous Calibration: A post-hoc model-agnostic framework for improved generalization [8.815439276597818]
  We introduce the notion of heterogeneous calibration that applies a post-hoc model-agnostic transformation to model outputs for improving AUC performance on binary classification tasks. We refer to simple patterns as heterogeneous partitions of the feature space and show theoretically that perfectly calibrating each partition separately optimize AUC. While the theoretical optimality of this framework holds for any model, we focus on deep neural networks (DNNs) and test the simplest instantiation of this paradigm on a variety of open-source datasets.
  arXiv  Detail & Related papers  (2022-02-10T05:08:50Z)
• Revisiting Consistency Regularization for Semi-Supervised Learning [80.28461584135967]
  We propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss. Experimental results show that our model defines a new state of the art for various datasets and settings.
  arXiv  Detail & Related papers  (2021-12-10T20:46:13Z)
• Adaptive Federated Optimization [43.78438670284309]
  In Federated learning, a large number of clients coordinate with a central server to learn a model without sharing their own data. adaptive optimization methods have notable success in combating such issues. We show that the use adaptives can significantly improve the performance of federated learning.
  arXiv  Detail & Related papers  (2020-02-29T16:37:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.