Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning

• URL: http://arxiv.org/abs/2311.11077v1
• Date: Sat, 18 Nov 2023 13:53:26 GMT
• Title: Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning
• Authors: Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engl\"ander, Timo Imhof, Ivan Vuli\'c, Sebastian Ruder, Iryna Gurevych, Jonas Pfeiffer
• Abstract summary: We introduce Adapters, an open-source library that unifies parameter-efficient and modular transfer learning in large language models. By integrating 10 diverse adapter methods into a unified interface, Adapters offers ease of use and flexible configuration.
• Score: 109.25673110120906
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We introduce Adapters, an open-source library that unifies parameter-efficient and modular transfer learning in large language models. By integrating 10 diverse adapter methods into a unified interface, Adapters offers ease of use and flexible configuration. Our library allows researchers and practitioners to leverage adapter modularity through composition blocks, enabling the design of complex adapter setups. We demonstrate the library's efficacy by evaluating its performance against full fine-tuning on various NLP tasks. Adapters provides a powerful tool for addressing the challenges of conventional fine-tuning paradigms and promoting more efficient and modular transfer learning. The library is available via https://adapterhub.ml/adapters.

Related papers

• MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair [5.006064616335817]
  Large Language Models (LLMs) have shown good performance in several software development-related tasks. This research proposes continual merging and empirically studies the capabilities of merged adapters in Code LLMs.
  arXiv  Detail & Related papers  (2024-08-18T18:45:48Z)
• Towards Modular LLMs by Building and Reusing a Library of LoRAs [64.43376695346538]
  We study how to best build a library of adapters given multi-task data. We introduce model-based clustering, MBC, a method that groups tasks based on the similarity of their adapter parameters. To re-use the library, we present a novel zero-shot routing mechanism, Arrow, which enables dynamic selection of the most relevant adapters.
  arXiv  Detail & Related papers  (2024-05-18T03:02:23Z)
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models [75.25782573728677]
  This paper presents a framework for adapter-based parameter-efficient fine-tuning (PEFT) of language models (LLMs) The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning.
  arXiv  Detail & Related papers  (2023-04-04T16:31:37Z)
• AdaMix: Mixture-of-Adapter for Parameter-efficient Tuning of Large Language Models [119.7093605087114]
  Fine-tuning large-scale pre-trained language models to downstream tasks require updating hundreds of millions of parameters. This not only increases the serving cost to store a large copy of the model weights for every task, but also exhibits instability during few-shot task adaptation. We introduce a new mechanism to improve adapter capacity without increasing parameters or computational cost by two key techniques.
  arXiv  Detail & Related papers  (2022-05-24T23:41:22Z)
• Adaptable Adapters [74.65986170056945]
  State-of-the-art pretrained NLP models contain a hundred million to trillion parameters. Adaptable adapters contain different activation functions for different layers and different input data. We show that adaptable adapters achieve on-par performances with the standard adapter architecture.
  arXiv  Detail & Related papers  (2022-05-03T14:59:27Z)
• AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters [34.86139827292556]
  Open-access dissemination of pretrained language models has led to a democratization of state-of-the-art natural language processing (NLP) research. This also allows people outside of NLP to use such models and adapt them to specific use-cases. In this work, we aim to overcome this gap by providing a tool which allows researchers to leverage pretrained models without writing a single line of code.
  arXiv  Detail & Related papers  (2021-08-18T11:56:01Z)
• What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning [15.204498046112194]
  We propose a data-driven adapter plugging strategy with Neural Architecture Search (NAS), which automatically determines where to plug for those adapter modules. We also propose a NAS-adapter module for adapter structure design in a NAS-driven learning scheme, which automatically discovers effective adapter module structures for different domains.
  arXiv  Detail & Related papers  (2020-07-24T09:12:37Z)
• AdapterHub: A Framework for Adapting Transformers [148.6877231725939]
  AdapterHub is a framework that allows dynamic "stitching-in" of pre-trained adapters for different tasks and languages. Our framework enables scalable and easy access to sharing of task-specific models.
  arXiv  Detail & Related papers  (2020-07-15T15:56:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.