Related papers: OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

URL: http://arxiv.org/abs/2307.03084v1
Date: Wed, 5 Jul 2023 16:30:14 GMT
Title: OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models
Authors: Shengding Hu, Ning Ding, Weilin Zhao, Xingtai Lv, Zhen Zhang, Zhiyuan Liu, Maosong Sun
Abstract summary: We present OpenDelta, an open-source library that overcomes limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs.
Score: 81.7855202178564
License: http://creativecommons.org/licenses/by/4.0/
Abstract: The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as "delta tuning", which updates only a small subset of parameters, known as "delta modules", while keeping the backbone model's parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.