FedDANE: A Federated Newton-Type Method

• URL: http://arxiv.org/abs/2001.01920v1
• Date: Tue, 7 Jan 2020 07:44:41 GMT
• Title: FedDANE: A Federated Newton-Type Method
• Authors: Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith
• Abstract summary: Federated learning aims to jointly learn low statistical models over massively distributed datasets. We propose FedDANE, an optimization that we adapt from DANE, to handle federated learning.
• Score: 49.9423212899788
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning aims to jointly learn statistical models over massively distributed remote devices. In this work, we propose FedDANE, an optimization method that we adapt from DANE, a method for classical distributed optimization, to handle the practical constraints of federated learning. We provide convergence guarantees for this method when learning over both convex and non-convex functions. Despite encouraging theoretical results, we find that the method has underwhelming performance empirically. In particular, through empirical simulations on both synthetic and real-world datasets, FedDANE consistently underperforms baselines of FedAvg and FedProx in realistic federated settings. We identify low device participation and statistical device heterogeneity as two underlying causes of this underwhelming performance, and conclude by suggesting several directions of future work.

Related papers

• Tighter Performance Theory of FedExProx [85.92481138826949]
  We revisit FedExProx - a recently proposed distributed optimization method designed to enhance convergence properties of parallel algorithms via extrapolation. We develop a novel analysis framework, establishing a tighter linear convergence rate for non-strongly convex quadratic problems. We extend the applicability of our analysis to general functions satisfying the Polyak-Lojasiewicz condition, outperforming the previous strongly convex analysis.
  arXiv  Detail & Related papers  (2024-10-20T11:53:25Z)
• Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques [65.55451717632317]
  We study Multi-Agent Reinforcement Learning from Human Feedback (MARLHF), exploring both theoretical foundations and empirical validations. We define the task as identifying Nash equilibrium from a preference-only offline dataset in general-sum games. Our findings underscore the multifaceted approach required for MARLHF, paving the way for effective preference-based multi-agent systems.
  arXiv  Detail & Related papers  (2024-09-01T13:14:41Z)
• On ADMM in Heterogeneous Federated Learning: Personalization, Robustness, and Fairness [16.595935469099306]
  We propose FLAME, an optimization framework by utilizing the alternating direction method of multipliers (ADMM) to train personalized and global models. Our theoretical analysis establishes the global convergence and two kinds of convergence rates for FLAME under mild assumptions. Our experimental findings show that FLAME outperforms state-of-the-art methods in convergence and accuracy, and it achieves higher test accuracy under various attacks.
  arXiv  Detail & Related papers  (2024-07-23T11:35:42Z)
• The Common Stability Mechanism behind most Self-Supervised Learning Approaches [64.40701218561921]
  We provide a framework to explain the stability mechanism of different self-supervised learning techniques. We discuss the working mechanism of contrastive techniques like SimCLR, non-contrastive techniques like BYOL, SWAV, SimSiam, Barlow Twins, and DINO. We formulate different hypotheses and test them using the Imagenet100 dataset.
  arXiv  Detail & Related papers  (2024-02-22T20:36:24Z)
• Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Data [6.024145412139383]
  Federated learning is an emerging distributed machine learning framework aiming at protecting data privacy. Data in different clients contain both common knowledge and personalized knowledge. We develop a novel personalized federated learning framework for heterogeneous data, which we refer to as FedSplit.
  arXiv  Detail & Related papers  (2023-12-07T13:05:47Z)
• FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation [95.85026305874824]
  We introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices. We conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiency.
  arXiv  Detail & Related papers  (2022-12-14T13:57:01Z)
• Faster Adaptive Federated Learning [84.38913517122619]
  Federated learning has attracted increasing attention with the emergence of distributed data. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
  arXiv  Detail & Related papers  (2022-12-02T05:07:50Z)
• Depersonalized Federated Learning: Tackling Statistical Heterogeneity by Alternating Stochastic Gradient Descent [6.394263208820851]
  Federated learning (FL) enables devices to train a common machine learning (ML) model for intelligent inference without data sharing. Raw data held by various cooperativelyicipators are always non-identically distributedly. We propose a new FL that can significantly statistical optimize by the de-speed of this process.
  arXiv  Detail & Related papers  (2022-10-07T10:30:39Z)
• Achieving Statistical Optimality of Federated Learning: Beyond Stationary Points [19.891597817559038]
  Federated Learning (FL) is a promising framework that has great potentials in privacy preservation and in lowering the computation load at the cloud. Recent work raised concerns on two methods: (1) their fixed points do not correspond to the stationary points of the original optimization problem, and (2) the common model found might not generalize well locally. We show, in the general kernel regression setting, that both FedAvg and FedProx converge to the minimax-optimal error rates.
  arXiv  Detail & Related papers  (2021-06-29T09:59:43Z)
• Improving Federated Relational Data Modeling via Basis Alignment and Weight Penalty [18.096788806121754]
  Federated learning (FL) has attracted increasing attention in recent years. We present a modified version of the graph neural network algorithm that performs federated modeling over Knowledge Graph (KG) We propose a novel optimization algorithm, named FedAlign, with 1) optimal transportation (OT) for on-client personalization and 2) weight constraint to speed up the convergence. Empirical results show that our proposed method outperforms the state-of-the-art FL methods, such as FedAVG and FedProx, with better convergence.
  arXiv  Detail & Related papers  (2020-11-23T12:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.