RE-Adapt: Reverse Engineered Adaptation of Large Language Models
- URL: http://arxiv.org/abs/2405.15007v1
- Date: Thu, 23 May 2024 19:23:40 GMT
- Title: RE-Adapt: Reverse Engineered Adaptation of Large Language Models
- Authors: William Fleshman, Benjamin Van Durme,
- Abstract summary: We introduce RE-Adapt, an approach to fine-tuning large language models on new domains without degrading any pre-existing instruction-tuning.
We reverse engineer an adapter which isolates what an instruction-tuned model has learned beyond its corresponding pretrained base model.
We can then fine-tune the base model on a new domain and readapt it to instruction following with the reverse engineered adapter.
- Score: 37.969478059005574
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We introduce RE-Adapt, an approach to fine-tuning large language models on new domains without degrading any pre-existing instruction-tuning. We reverse engineer an adapter which isolates what an instruction-tuned model has learned beyond its corresponding pretrained base model. Importantly, this requires no additional data or training. We can then fine-tune the base model on a new domain and readapt it to instruction following with the reverse engineered adapter. RE-Adapt and our low-rank variant LoRE-Adapt both outperform other methods of fine-tuning, across multiple popular LLMs and datasets, even when the models are used in conjunction with retrieval-augmented generation.
Related papers
- Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusion [10.322832012497722]
Class-incremental learning is a challenging problem, where the goal is to train a model that can classify data from an increasing number of classes over time.
With the advancement of vision-language pre-trained models such as CLIP, they demonstrate good generalization ability.
However, further adaptation to downstream tasks by simply fine-tuning the model leads to severe forgetting.
Most existing works with pre-trained models assume that the forgetting of old classes is uniform when the model acquires new knowledge.
arXiv Detail & Related papers (2024-07-19T09:20:33Z) - RE-AdaptIR: Improving Information Retrieval through Reverse Engineered Adaptation [37.969478059005574]
Large language models (LLMs) fine-tuned for text-retrieval have demonstrated state-of-the-art results across several information retrieval benchmarks.
We explore the effectiveness of extending reverse engineered adaptation to the context of information retrieval.
arXiv Detail & Related papers (2024-06-20T22:28:11Z) - Do Not Wait: Learning Re-Ranking Model Without User Feedback At Serving Time in E-Commerce [16.316227411757797]
We propose a novel extension of online learning methods for re-ranking modeling, which we term LAST.
It circumvents the requirement of user feedback by using a surrogate model to provide the instructional signal needed to steer model improvement.
LAST can be seamlessly integrated into existing online learning systems to create a more adaptive and responsive recommendation experience.
arXiv Detail & Related papers (2024-06-20T05:15:48Z) - RaFe: Ranking Feedback Improves Query Rewriting for RAG [83.24385658573198]
We propose a framework for training query rewriting models free of annotations.
By leveraging a publicly available reranker, oursprovides feedback aligned well with the rewriting objectives.
arXiv Detail & Related papers (2024-05-23T11:00:19Z) - Direct Language Model Alignment from Online AI Feedback [78.40436231613754]
Direct alignment from preferences (DAP) methods have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF)
In this study, we posit that online feedback is key and improves DAP methods.
Our method, online AI feedback (OAIF) uses an LLM as annotator: on each training, we sample two responses from the current model and prompt the LLM annotator to choose which one is preferred, thus providing online feedback.
arXiv Detail & Related papers (2024-02-07T12:31:13Z) - Pluggable Neural Machine Translation Models via Memory-augmented Adapters [25.26982333390014]
We propose a memory-augmented adapter to steer pretrained NMT models in a pluggable manner.
Specifically, we construct a multi-granular memory based on the user-provided text samples.
We also propose a training strategy using memory dropout to reduce spurious dependencies between the NMT model and the memory.
arXiv Detail & Related papers (2023-07-12T09:23:41Z) - $BT^2$: Backward-compatible Training with Basis Transformation [107.37014712361788]
Retrieval system often requires recomputing the representation of every piece of data in the gallery when updating to a better representation model.
This process is known as backfilling and can be especially costly in the real world where the gallery often contains billions of samples.
Recently, researchers have proposed the idea of Backward compatible Training (BCT) where the new representation model can be trained with an auxiliary loss to make it backward compatible with the old representation.
arXiv Detail & Related papers (2022-11-08T04:00:23Z) - Re-parameterizing Your Optimizers rather than Architectures [119.08740698936633]
We propose a novel paradigm of incorporating model-specific prior knowledge into Structurals and using them to train generic (simple) models.
As an implementation, we propose a novel methodology to add prior knowledge by modifying the gradients according to a set of model-specific hyper- parameters.
For a simple model trained with a Repr, we focus on a VGG-style plain model and showcase that such a simple model trained with a Repr, which is referred to as Rep-VGG, performs on par with the recent well-designed models.
arXiv Detail & Related papers (2022-05-30T16:55:59Z) - Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay [49.691610143011566]
We propose two novel knowledge transfer techniques for class-incremental learning (CIL)
First, we propose data-free generative replay (DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic samples from a generative model.
Second, we introduce dual-teacher information distillation (DT-ID) for knowledge distillation from two teachers to one student.
arXiv Detail & Related papers (2021-06-17T22:13:15Z) - Adaptable Multi-Domain Language Model for Transformer ASR [16.8397357399749]
The proposed model can reuse the full fine-tuned LM which is fine-tuned using all layers of an original model.
The proposed model is also effective in reducing the model maintenance cost because it is possible to omit the costly and time-consuming common LM pre-training process.
arXiv Detail & Related papers (2020-08-14T06:33:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.