lo-fi: distributed fine-tuning without communication

• URL: http://arxiv.org/abs/2210.11948v1
• Date: Wed, 19 Oct 2022 20:15:18 GMT
• Title: lo-fi: distributed fine-tuning without communication
• Abstract summary: We investigate completely local fine-tuning, which we refer to as lo-fi. During lo-fi, each node is fine-tuned independently without any communication. By removing the communication requirement, lo-fi reduces resource barriers for fine-tuning large models.
• Score: 61.87390899015787
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When fine-tuning large neural networks, it is common to use multiple nodes and to communicate gradients at each optimization step. By contrast, we investigate completely local fine-tuning, which we refer to as lo-fi. During lo-fi, each node is fine-tuned independently without any communication. Then, the weights are averaged across nodes at the conclusion of fine-tuning. When fine-tuning DeiT-base and DeiT-large on ImageNet, this procedure matches accuracy in-distribution and improves accuracy under distribution shift compared to the baseline, which observes the same amount of data but communicates gradients at each step. We also observe that lo-fi matches the baseline's performance when fine-tuning OPT language models (up to 1.3B parameters) on Common Crawl. By removing the communication requirement, lo-fi reduces resource barriers for fine-tuning large models and enables fine-tuning in settings with prohibitive communication cost.

Related papers

• Biased Federated Learning under Wireless Heterogeneity [7.3716675761469945]
  Federated learning (FL) is a promising framework for computation, enabling collaborative model training without sharing private data. Existing wireless computation works primarily adopt two communication strategies: (1) over-the-air (OTA) which exploits wireless signal superposition, and (2) over-the-air (OTA) which allocates resources for convergence. This paper proposes novel OTA and digital FL updates that allow a structured, time-in-place bias, thereby reducing variance in FL updates.
  arXiv  Detail & Related papers  (2025-03-08T05:55:14Z)
• Noise-Robust and Resource-Efficient ADMM-based Federated Learning [6.957420925496431]
  Federated learning (FL) leverages client-server communications to train global models on decentralized data. We propose a novel FL algorithm that enhances robustness against communication noise while also reducing communication load.
  arXiv  Detail & Related papers  (2024-09-20T12:32:22Z)
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. Experiments show that SpaFL improves accuracy while requiring much less communication and computing resources compared to sparse baselines.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• Decentralized Federated Learning Over Imperfect Communication Channels [68.08499874460857]
  This paper analyzes the impact of imperfect communication channels on decentralized federated learning (D-FL) It determines the optimal number of local aggregations per training round, adapting to the network topology and imperfect channels. It is seen that D-FL, with an optimal number of local aggregations, can outperform its potential alternatives by over 10% in training accuracy.
  arXiv  Detail & Related papers  (2024-05-21T16:04:32Z)
• FedNAR: Federated Optimization with Normalized Annealing Regularization [54.42032094044368]
  We explore the choices of weight decay and identify that weight decay value appreciably influences the convergence of existing FL algorithms. We develop Federated optimization with Normalized Annealing Regularization (FedNAR), a plug-in that can be seamlessly integrated into any existing FL algorithms.
  arXiv  Detail & Related papers  (2023-10-04T21:11:40Z)
• Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Training [6.557328947642343]
  Distributed full-graph training of Graph Neural Networks (GNNs) over large graphs is bandwidth-demanding and time-consuming. This paper proposes an efficient GNN training system, AdaQP, to expedite distributed full-graph training.
  arXiv  Detail & Related papers  (2023-06-02T09:02:09Z)
• Distributed Learning over Networks with Graph-Attention-Based Personalization [49.90052709285814]
  We propose a graph-based personalized algorithm (GATTA) for distributed deep learning. In particular, the personalized model in each agent is composed of a global part and a node-specific part. By treating each agent as one node in a graph the node-specific parameters as its features, the benefits of the graph attention mechanism can be inherited.
  arXiv  Detail & Related papers  (2023-05-22T13:48:30Z)
• On Convergence of Federated Averaging Langevin Dynamics [22.013125418713763]
  We propose a federated averaging Langevin algorithm (FA-LD) for uncertainty quantification and mean predictions with distributed clients. We develop theoretical guarantees for FA-LD for strongly log-con distributions with non-icaved data. We show convergence results based on different averaging schemes where only partial device updates are available.
  arXiv  Detail & Related papers  (2021-12-09T18:54:29Z)
• Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning [75.45968495410047]
  Communication of model updates between client nodes and the central aggregating server is a major bottleneck in federated learning. Gradient quantization is an effective way of reducing the number of bits required to communicate each model update. We propose an adaptive quantization strategy called AdaFL that aims to achieve communication efficiency as well as a low error floor.
  arXiv  Detail & Related papers  (2021-02-08T19:14:21Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)
• PushNet: Efficient and Adaptive Neural Message Passing [1.9121961872220468]
  Message passing neural networks have recently evolved into a state-of-the-art approach to representation learning on graphs. Existing methods perform synchronous message passing along all edges in multiple subsequent rounds. We consider a novel asynchronous message passing approach where information is pushed only along the most relevant edges until convergence.
  arXiv  Detail & Related papers  (2020-03-04T18:15:30Z)
• Adaptive Propagation Graph Convolutional Network [17.41698818541144]
  Graph convolutional networks (GCNs) are a family of neural network models that perform inference on graph data. We show that state-of-the-art results can be achieved by adapting the number of communication steps independently at every node. We show that the proposed adaptive propagation GCN (AP-GCN) achieves superior or similar results to the best proposed models.
  arXiv  Detail & Related papers  (2020-02-24T15:31:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.