Effective LoRA Adapter Routing using Task Representations

• URL: http://arxiv.org/abs/2601.21795v2
• Date: Fri, 30 Jan 2026 09:21:58 GMT
• Title: Effective LoRA Adapter Routing using Task Representations
• Authors: Akash Dhasade, Anne-Marie Kermarrec, Igor Pavlovic, Diana Petrescu, Rafael Pires, Mathis Randl, Martijn de Vos,
• Abstract summary: Low-rank adaptation (LoRA) enables parameter efficient specialization of large language models (LLMs) through modular adapters.<n>We introduce LORAUTER, a novel routing framework that selects and composes LoRA adapters using task representations rather than adapter characteristics.
• Score: 3.0111172730438565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-rank adaptation (LoRA) enables parameter efficient specialization of large language models (LLMs) through modular adapters, resulting in rapidly growing public adapter pools spanning diverse tasks. Effectively using these adapters requires routing: selecting and composing the appropriate adapters for a query. We introduce LORAUTER, a novel routing framework that selects and composes LoRA adapters using task representations rather than adapter characteristics. Unlike existing approaches that map queries directly to adapters, LORAUTER routes queries via task embeddings derived from small validation sets and does not require adapter training data. By operating at the task level, LORAUTER achieves efficient routing that scales with the number of tasks rather than the number of adapters. Experiments across multiple tasks show that LORAUTER consistently outperforms baseline routing approaches, matching Oracle performance (101.2%) when task-aligned adapters exist and achieving state-of-the-art results on unseen tasks (+5.2 points). We further demonstrate the robustness of LORAUTER to very large, noisy adapter pools by scaling it to over 1500 adapters.

Related papers

• Data-driven Clustering and Merging of Adapters for On-device Large Language Models [34.58968471192321]
  On-device large language models commonly employ task-specific adapters (e.g., LoRAs) to deliver strong performance on downstream tasks.<n>This raises a critical challenge: how to select representative adapters that generalize well across multiple tasks.<n>We propose a novel method D2C for adapter clustering that leverages minimal task-specific examples and employs an iterative optimization process to refine cluster assignments.
  arXiv  Detail & Related papers  (2026-01-24T12:25:46Z)
• Towards Optimal Adapter Placement for Efficient Transfer Learning [73.1149084352343]
  PETL aims to adapt pre-trained models to new downstream tasks while minimizing the number of fine-tuned parameters. adapters, a popular approach in PETL, inject additional capacity into existing networks by incorporating low-rank projections. This paper investigates the relationship between the placement of an adapter and its performance.
  arXiv  Detail & Related papers  (2024-10-21T10:37:17Z)
• MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair [5.006064616335817]
  Large Language Models (LLMs) have shown high capabilities in several software development-related tasks.<n> adapters offer a more efficient way to customize LLMs for particular needs.<n>Model (and adapter) merging have emerged as a technique to develop one model capable of multiple tasks.
  arXiv  Detail & Related papers  (2024-08-18T18:45:48Z)
• Towards Modular LLMs by Building and Reusing a Library of LoRAs [64.43376695346538]
  We study how to best build a library of adapters given multi-task data. We introduce model-based clustering, MBC, a method that groups tasks based on the similarity of their adapter parameters. To re-use the library, we present a novel zero-shot routing mechanism, Arrow, which enables dynamic selection of the most relevant adapters.
  arXiv  Detail & Related papers  (2024-05-18T03:02:23Z)
• Stylus: Automatic Adapter Selection for Diffusion Models [81.90482700433822]
  We introduce Stylus, which efficiently selects and automatically composes task-specific adapters based on a prompt's keywords. Stylus outlines a three-stage approach that first summarizes adapters with improved descriptions and embeddings, retrieves relevant adapters, and then further assembles adapters based on prompts' keywords by checking how well they fit the prompt.
  arXiv  Detail & Related papers  (2024-04-29T17:59:16Z)
• Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models [12.230087530720652]
  We introduce an adapter module that has a better efficiency in large scale multi-task adaptation scenario. The adapter consists of a single shared controller network and multiple task-level adapter heads.
  arXiv  Detail & Related papers  (2024-03-25T17:21:56Z)
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models [75.25782573728677]
  This paper presents a framework for adapter-based parameter-efficient fine-tuning (PEFT) of language models (LLMs) The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning.
  arXiv  Detail & Related papers  (2023-04-04T16:31:37Z)
• Multi-Head Adapter Routing for Cross-Task Generalization [56.75667096355806]
  Polytropon learns an inventory of adapters and a routing function that selects a subset of adapters for each task during both pre-training and few-shot adaptation. We find that routing is most beneficial during multi-task pre-training rather than during few-shot adaptation.
  arXiv  Detail & Related papers  (2022-11-07T19:35:55Z)
• Adaptable Adapters [74.65986170056945]
  State-of-the-art pretrained NLP models contain a hundred million to trillion parameters. Adaptable adapters contain different activation functions for different layers and different input data. We show that adaptable adapters achieve on-par performances with the standard adapter architecture.
  arXiv  Detail & Related papers  (2022-05-03T14:59:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.