LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed
Tasks in the Wild
- URL: http://arxiv.org/abs/2402.09997v1
- Date: Thu, 15 Feb 2024 15:02:46 GMT
- Title: LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed
Tasks in the Wild
- Authors: Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang,
Kun Kuang, Fei Wu
- Abstract summary: Low-Rank Adaptation (LoRA) provides an efficient solution for fine-tuning large language models (LLM)
LoraRetriever is a retrieve-then-compose framework that adaptively retrieves and composes multiple LoRAs according to the input prompts.
Experimental results indicate that LoraRetriever consistently outperforms the baselines.
- Score: 76.67343971195267
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Low-Rank Adaptation (LoRA) provides an effective yet efficient solution for
fine-tuning large language models (LLM). The modular and plug-and-play nature
of LoRA enables the integration of diverse domain-specific LoRAs to enhance the
capabilities of LLMs. Previous research on exploiting multiple LoRAs either
focuses on specific isolated downstream tasks or fixes the selection of LoRAs
during training. However, in real-world scenarios, LLMs receive diverse prompts
covering different tasks, and the pool of candidate LoRAs is often dynamically
updated. To bridge this gap, we propose LoraRetriever, a retrieve-then-compose
framework that adaptively retrieves and composes multiple LoRAs according to
the input prompts. LoraRetriever contains three main components: firstly,
identifying and retrieving LoRAs relevant to the given input; secondly,
formulating strategies for effectively integrating the retrieved LoRAs; and
thirdly, developing efficient batch inference to accommodate heterogeneous
requests. Experimental results indicate that LoraRetriever consistently
outperforms the baselines, highlighting its practical effectiveness and
versatility.
Related papers
- Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning [57.36978335727009]
Low-Rank Adaptation (LoRA) offers an efficient way to fine-tune large language models (LLMs)
In this paper, we propose a framework that adaptively retrieves and composes multiple LoRAs based on input prompts.
arXiv Detail & Related papers (2024-06-24T05:24:41Z) - MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models [4.978361907192563]
We introduce MeteoRA, a scalable multi-knowledge LoRA fusion framework designed for large language models (LLMs)
MeteoRA integrates various LoRA adapters in a Mixture-of-Experts (MoE) style into the base LLM, enabling the model to automatically select the most pertinent adapter based on the task input.
Our evaluations, featuring the LlaMA2-13B and LlaMA3-8B base models equipped with off-the-shelf 28 LoRA adapters through MeteoRA, demonstrate equivalent performance with the individual adapters.
arXiv Detail & Related papers (2024-05-19T20:46:07Z) - Mixture of LoRA Experts [87.50120181861362]
This paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection.
The MoLE approach achieves superior LoRA fusion performance in comparison to direct arithmetic merging.
arXiv Detail & Related papers (2024-04-21T11:59:53Z) - Multi-LoRA Composition for Image Generation [111.88506763476249]
We study multi-LoRA composition through a decoding-centric perspective.
We present two training-free methods: LoRA Switch, which alternates between different LoRAs at each denoising step, and LoRA Composite, which simultaneously incorporates all LoRAs to guide more cohesive image synthesis.
arXiv Detail & Related papers (2024-02-26T18:59:18Z) - Multimodal Instruction Tuning with Conditional Mixture of LoRA [54.65520214291653]
This paper introduces a novel approach that integrates multimodal instruction tuning with Low-Rank Adaption (LoRA)
It innovates upon LoRA by dynamically constructing low-rank adaptation matrices tailored to the unique demands of each input instance.
Experimental results on various multimodal evaluation datasets indicate that MixLoRA not only outperforms the conventional LoRA with the same or even higher ranks.
arXiv Detail & Related papers (2024-02-24T20:15:31Z) - LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative
Tasks [72.88244322513039]
LoRA employs lightweight modules to customize large language models (LLMs) for each downstream task or domain.
We propose LoRA-Flow, which utilizes dynamic weights to adjust the impact of different LoRAs.
Experiments across six generative tasks demonstrate that our method consistently outperforms baselines with task-level fusion weights.
arXiv Detail & Related papers (2024-02-18T04:41:25Z) - MultiLoRA: Democratizing LoRA for Better Multi-Task Learning [20.750808913757396]
LoRA achieves remarkable resource efficiency and comparable performance when adapting LLMs for specific tasks.
LoRA is dominated by a small number of top singular vectors while fine-tuning decomposes into a set of less important unitary transforms.
We propose MultiLoRA for better multi-task adaptation by reducing the dominance of top singular vectors observed in LoRA.
arXiv Detail & Related papers (2023-11-20T02:59:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.