Statistical Estimation and Inference via Local SGD in Federated Learning

• URL: http://arxiv.org/abs/2109.01326v1
• Date: Fri, 3 Sep 2021 06:02:19 GMT
• Title: Statistical Estimation and Inference via Local SGD in Federated Learning
• Authors: Xiang Li, Jiadong Liang, Xiangyu Chang, Zhihua Zhang
• Abstract summary: This paper studies how to perform statistical estimation and inference in the federated setting. We analyze the so-called Local SGD, a multi-round estimation procedure that uses intermittent communication to improve communication efficiency.
• Score: 23.32304977333178
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) makes a large amount of edge computing devices (e.g., mobile phones) jointly learn a global model without data sharing. In FL, data are generated in a decentralized manner with high heterogeneity. This paper studies how to perform statistical estimation and inference in the federated setting. We analyze the so-called Local SGD, a multi-round estimation procedure that uses intermittent communication to improve communication efficiency. We first establish a {\it functional central limit theorem} that shows the averaged iterates of Local SGD weakly converge to a rescaled Brownian motion. We next provide two iterative inference methods: the {\it plug-in} and the {\it random scaling}. Random scaling constructs an asymptotically pivotal statistic for inference by using the information along the whole Local SGD path. Both the methods are communication efficient and applicable to online data. Our theoretical and empirical results show that Local SGD simultaneously achieves both statistical efficiency and communication efficiency.

Related papers

• Fast-Convergent and Communication-Alleviated Heterogeneous Hierarchical Federated Learning in Autonomous Driving [29.646749372031593]
  Street Scene Semantic Understanding (denoted as TriSU) is a complex task for autonomous driving (AD) Inference model trained from data in a particular geographical region faces poor generalization when applied in other regions due to inter-city data domain-shift. Hierarchical Federated Learning (HFL) offers a potential solution for improving TriSU model generalization by collaborative privacy-preserving training over distributed datasets from different cities.
  arXiv  Detail & Related papers  (2024-09-29T05:27:40Z)
• Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models [54.02863371927658]
  Large Language Models (LLMs) have become indispensable in numerous real-world applications. Ferret is the first first-order method with shared randomness. It achieves high computational efficiency, reduced communication overhead, and fast convergence.
  arXiv  Detail & Related papers  (2024-09-10T07:28:13Z)
• FedLog: Personalized Federated Classification with Less Communication and More Flexibility [24.030147353437382]
  Federated representation learning (FRL) aims to learn personalized federated models with effective feature extraction from local data. To reduce the overhead, we propose to share sufficient data summaries instead of raw model parameters.
  arXiv  Detail & Related papers  (2024-07-11T09:40:29Z)
• Asynchronous Local Computations in Distributed Bayesian Learning [8.516532665507835]
  We propose gossip-based communication to leverage fast computations and reduce communication overhead simultaneously. We observe faster initial convergence and improved performance accuracy, especially in the low data range. We achieve on average 78% and over 90% classification accuracy respectively on the Gamma Telescope and mHealth data sets from the UCI ML repository.
  arXiv  Detail & Related papers  (2023-11-06T20:11:41Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• A Newton-type algorithm for federated learning based on incremental Hessian eigenvector sharing [5.404315085380945]
  We present an original communication-constrained Newton-type (NT) algorithm designed to accelerate Federated Learning (FL) The proposed solution is thoroughly validated on real datasets.
  arXiv  Detail & Related papers  (2022-02-11T17:52:56Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)
• Clustered Federated Learning via Generalized Total Variation Minimization [83.26141667853057]
  We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure. Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV) Our main algorithmic contribution is a fully decentralized federated learning algorithm.
  arXiv  Detail & Related papers  (2021-05-26T18:07:19Z)
• Improving Generative Adversarial Networks with Local Coordinate Coding [150.24880482480455]
  Generative adversarial networks (GANs) have shown remarkable success in generating realistic data from some predefined prior distribution. In practice, semantic information might be represented by some latent distribution learned from data. We propose an LCCGAN model with local coordinate coding (LCC) to improve the performance of generating data.
  arXiv  Detail & Related papers  (2020-07-28T09:17:50Z)
• FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data [59.50904660420082]
  Federated Learning (FL) has become a popular paradigm for learning from distributed data. To effectively utilize data at different devices without moving them to the cloud, algorithms such as the Federated Averaging (FedAvg) have adopted a "computation then aggregation" (CTA) model.
  arXiv  Detail & Related papers  (2020-05-22T23:07:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.