Good Intentions: Adaptive Parameter Management via Intent Signaling

• URL: http://arxiv.org/abs/2206.00470v4
• Date: Thu, 17 Aug 2023 15:17:53 GMT
• Title: Good Intentions: Adaptive Parameter Management via Intent Signaling
• Authors: Alexander Renz-Wieland, Andreas Kieslinger, Robert Gericke, Rainer Gemulla, Zoi Kaoudi, Volker Markl
• Abstract summary: We propose a novel intent signaling mechanism that integrates naturally into existing machine learning stacks. We then describe AdaPM, a fully adaptive, zero-tuning parameter manager based on this mechanism. In our evaluation, AdaPM matched or outperformed state-of-the-art parameter managers out of the box.
• Score: 50.01012642343155
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Parameter management is essential for distributed training of large machine learning (ML) tasks. Some ML tasks are hard to distribute because common approaches to parameter management can be highly inefficient. Advanced parameter management approaches -- such as selective replication or dynamic parameter allocation -- can improve efficiency, but to do so, they typically need to be integrated manually into each task's implementation and they require expensive upfront experimentation to tune correctly. In this work, we explore whether these two problems can be avoided. We first propose a novel intent signaling mechanism that integrates naturally into existing ML stacks and provides the parameter manager with crucial information about parameter accesses. We then describe AdaPM, a fully adaptive, zero-tuning parameter manager based on this mechanism. In contrast to prior systems, this approach separates providing information (simple, done by the task) from exploiting it effectively (hard, done automatically by AdaPM). In our experimental evaluation, AdaPM matched or outperformed state-of-the-art parameter managers out of the box, suggesting that automatic parameter management is possible.

Related papers

• Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition [16.647413058592125]
  Multi-level implicit discourse relation recognition (MIDRR) aims at identifying hierarchical discourse relations among arguments. In this paper, we propose a prompt-based. Efficient Multi-level IDRR (PEMI) framework to solve the above problems.
  arXiv  Detail & Related papers  (2024-02-23T03:53:39Z)
• Parameter-Efficient Fine-Tuning without Introducing New Latency [7.631596468553607]
  We introduce a novel adapter technique that directly applies the adapter to pre-trained parameters instead of the hidden representation. Our proposed method attains a new state-of-the-art outcome in terms of both performance and storage efficiency, storing only 0.03% parameters of full fine-tuning.
  arXiv  Detail & Related papers  (2023-05-26T08:44:42Z)
• Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning [91.5113227694443]
  We propose a novel visual. sensuous-aware fine-Tuning (SPT) scheme. SPT allocates trainable parameters to task-specific important positions. Experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing PEFT methods.
  arXiv  Detail & Related papers  (2023-03-15T12:34:24Z)
• Parameter-Efficient Low-Resource Dialogue State Tracking by Prompt Tuning [57.01260458860375]
  Dialogue state tracking (DST) is an important step in dialogue management to keep track of users' beliefs. Existing works fine-tune all language model (LM) parameters to tackle the DST task. We propose to use soft prompt token embeddings to learn task properties.
  arXiv  Detail & Related papers  (2023-01-26T03:01:59Z)
• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning [19.201899503691266]
  We measure the task dominance degree of a parameter by the total updates of each task on this parameter. We propose a Task-wise Adaptive learning rate approach, AdaTask, to separate the emphaccumulative gradients and hence the learning rate of each task. Experiments on computer vision and recommender system MTL datasets demonstrate that AdaTask significantly improves the performance of dominated tasks.
  arXiv  Detail & Related papers  (2022-11-28T04:24:38Z)
• Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attention [53.72897232951918]
  We propose adapters that utilize the kernel structure in self-attention to guide the assignment of tunable parameters. Our results show that our proposed adapters can attain or improve the strong performance of existing baselines.
  arXiv  Detail & Related papers  (2022-05-07T20:52:54Z)
• Replicate or Relocate? Non-Uniform Access in Parameter Servers [74.89066750738971]
  We present Lapse2, a PS that replicates hot spot parameters, relocates less frequently accessed parameters, and employs specialized techniques to manage nondeterminism. In our experimental study, Lapse2 outperformed existing, single-technique PSs by up to one order of magnitude.
  arXiv  Detail & Related papers  (2021-04-01T14:52:32Z)
• Dynamic Parameter Allocation in Parameter Servers [74.250687861348]
  We propose to integrate dynamic parameter allocation into parameter servers, describe an efficient implementation of such a parameter server called Lapse. We found that Lapse provides near-linear scaling and can be orders of magnitude faster than existing parameter servers.
  arXiv  Detail & Related papers  (2020-02-03T11:37:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.