ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuning

• URL: http://arxiv.org/abs/2603.10160v1
• Date: Tue, 10 Mar 2026 18:51:27 GMT
• Title: ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuning
• Abstract summary: Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks.<n>Existing Mixture-of-LoRAs routers assign a learned routing weight to each LoRA to enable end-to-end training of the router.<n>We propose a new router designed that we call Reinforcement Routing for Mixture-of-LoRAs (ReMix)
• Score: 85.39146836671897
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks. Mixture-of-LoRAs models expand neural networks efficiently by routing each layer input to a small subset of specialized LoRAs of the layer. Existing Mixture-of-LoRAs routers assign a learned routing weight to each LoRA to enable end-to-end training of the router. Despite their empirical promise, we observe that the routing weights are typically extremely imbalanced across LoRAs in practice, where only one or two LoRAs often dominate the routing weights. This essentially limits the number of effective LoRAs and thus severely hinders the expressive power of existing Mixture-of-LoRAs models. In this work, we attribute this weakness to the nature of learnable routing weights and rethink the fundamental design of the router. To address this critical issue, we propose a new router designed that we call Reinforcement Routing for Mixture-of-LoRAs (ReMix). Our key idea is using non-learnable routing weights to ensure all active LoRAs to be equally effective, with no LoRA dominating the routing weights. However, our routers cannot be trained directly via gradient descent due to our non-learnable routing weights. Hence, we further propose an unbiased gradient estimator for the router by employing the reinforce leave-one-out (RLOO) technique, where we regard the supervision loss as the reward and the router as the policy in reinforcement learning. Our gradient estimator also enables to scale up training compute to boost the predictive performance of our ReMix. Extensive experiments demonstrate that our proposed ReMix significantly outperform state-of-the-art parameter-efficient finetuning methods under a comparable number of activated parameters.

Related papers

• Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs [27.71932231964013]
  We propose a framework for composing frozen reasoning LoRA experts through unit-scale hard selection.<n>Experiments show that Hard-Routed MoR-LoRA preserves expert behavior while requiring substantially fewer trainable parameters than soft-routing mixture baselines.
  arXiv  Detail & Related papers  (2026-06-30T09:40:38Z)
• PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning [40.05865857566415]
  We propose Predictive Routing Replay (PR2), which augments each router with a lightweight evolution predictor that learns to anticipate short-horizon router evolution.<n>PR2 reduces routing-induced mismatch, improves RL stability, and yields stronger performance across various reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-05-29T22:28:08Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models [22.457766373989365]
  Low-Rank Adapters (LoRAs) have been substantially adopted across various fields, including instruction tuning and domain adaptation.<n>To address the limited expressive capacity of LoRA, the Mixture-of-Expert (MoE) has been introduced for incorporating multiple LoRA adapters.<n>We propose a new training strategy for MoE-LoRA, to stabilize and boost its feature learning procedure by multi-space projections.
  arXiv  Detail & Related papers  (2025-02-20T05:58:53Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning [9.91790333647256]
  Low-rank adaptation (LoRA) and its mixture-of-experts (MOE) variants are highly effective parameter-efficient fine-tuning (PEFT) methods. We propose Mixture of Low-Rank Adaptation (MiLoRA), a novel and efficient LoRA variant. MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism.
  arXiv  Detail & Related papers  (2024-10-23T17:04:40Z)
• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts [3.6301530893494127]
  MixLoRA is an approach to construct a resource-efficient sparse MoE model based on LoRA. Our evaluations show that MixLoRA improves about 9% accuracy compared to state-of-the-art PEFT methods in multi-task learning scenarios.
  arXiv  Detail & Related papers  (2024-04-22T02:15:52Z)
• ResLoRA: Identity Residual Mapping in Low-Rank Adaption [96.59370314485074]
  We propose ResLoRA, an improved framework of low-rank adaptation (LoRA) Our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-02-28T04:33:20Z)
• DoRA: Weight-Decomposed Low-Rank Adaptation [57.68678247436207]
  We introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA) DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning.
  arXiv  Detail & Related papers  (2024-02-14T17:59:34Z)
• Run LoRA Run: Faster and Lighter LoRA Implementations [50.347242693025336]
  LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This paper presents the RunLoRA framework for efficient implementations of LoRA. Experiments show up to 28% speedup on language modeling networks.
  arXiv  Detail & Related papers  (2023-12-06T10:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.