Dynamic Parameter Allocation in Parameter Servers

• URL: http://arxiv.org/abs/2002.00655v3
• Date: Fri, 3 Jul 2020 12:52:13 GMT
• Title: Dynamic Parameter Allocation in Parameter Servers
• Authors: Alexander Renz-Wieland, Rainer Gemulla, Steffen Zeuch, Volker Markl
• Abstract summary: We propose to integrate dynamic parameter allocation into parameter servers, describe an efficient implementation of such a parameter server called Lapse. We found that Lapse provides near-linear scaling and can be orders of magnitude faster than existing parameter servers.
• Score: 74.250687861348
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To keep up with increasing dataset sizes and model complexity, distributed training has become a necessity for large machine learning tasks. Parameter servers ease the implementation of distributed parameter management---a key concern in distributed training---, but can induce severe communication overhead. To reduce communication overhead, distributed machine learning algorithms use techniques to increase parameter access locality (PAL), achieving up to linear speed-ups. We found that existing parameter servers provide only limited support for PAL techniques, however, and therefore prevent efficient training. In this paper, we explore whether and to what extent PAL techniques can be supported, and whether such support is beneficial. We propose to integrate dynamic parameter allocation into parameter servers, describe an efficient implementation of such a parameter server called Lapse, and experimentally compare its performance to existing parameter servers across a number of machine learning tasks. We found that Lapse provides near-linear scaling and can be orders of magnitude faster than existing parameter servers.

Related papers

• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. Experiments show that SpaFL improves accuracy while requiring much less communication and computing resources compared to sparse baselines.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning [30.251155072822055]
  Prototype-based HyperAdapter (PHA) is a novel framework built on the adapter-tuning and hypernetwork. It introduces an instance-dense retriever and prototypical hypernetwork to generate conditional modules in a sample-efficient manner. We show that PHA strikes a better trade-off between trainable parameters, accuracy on stream tasks, and sample efficiency.
  arXiv  Detail & Related papers  (2023-10-18T02:42:17Z)
• Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning [91.5113227694443]
  We propose a novel visual. sensuous-aware fine-Tuning (SPT) scheme. SPT allocates trainable parameters to task-specific important positions. Experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing PEFT methods.
  arXiv  Detail & Related papers  (2023-03-15T12:34:24Z)
• Architecting Peer-to-Peer Serverless Distributed Machine Learning Training for Improved Fault Tolerance [1.495380389108477]
  Serverless computing is a new paradigm for cloud computing that uses functions as a computational unit. By distributing the workload, distributed machine learning can speed up the training process and allow more complex models to be trained. We propose exploring the use of serverless computing in distributed machine learning training and comparing the performance of P2P architecture with the parameter server architecture.
  arXiv  Detail & Related papers  (2023-02-27T17:38:47Z)
• Parameter-Efficient Low-Resource Dialogue State Tracking by Prompt Tuning [57.01260458860375]
  Dialogue state tracking (DST) is an important step in dialogue management to keep track of users' beliefs. Existing works fine-tune all language model (LM) parameters to tackle the DST task. We propose to use soft prompt token embeddings to learn task properties.
  arXiv  Detail & Related papers  (2023-01-26T03:01:59Z)
• PiPar: Pipeline Parallelism for Collaborative Machine Learning [16.131285496487678]
  Collaborative machine learning (CML) techniques have been proposed to train deep learning models across multiple mobile devices and a server. CML techniques are privacy-preserving as a local model that is trained on each device instead of the raw data from the device is shared with the server. We identify idling resources on the server and devices due to sequential computation and communication as the principal cause of low resource utilization.
  arXiv  Detail & Related papers  (2022-12-01T20:51:47Z)
• Replicate or Relocate? Non-Uniform Access in Parameter Servers [74.89066750738971]
  We present Lapse2, a PS that replicates hot spot parameters, relocates less frequently accessed parameters, and employs specialized techniques to manage nondeterminism. In our experimental study, Lapse2 outperformed existing, single-technique PSs by up to one order of magnitude.
  arXiv  Detail & Related papers  (2021-04-01T14:52:32Z)
• Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning [73.82875010696849]
  Machine learning algorithms are deployed at the network edge for training artificial intelligence (AI) models. This paper focuses on the novel joint design of parameter (computation load) allocation and bandwidth allocation.
  arXiv  Detail & Related papers  (2020-03-10T05:52:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.