Crowded in B-Space: Calibrating Shared Directions for LoRA Merging

• URL: http://arxiv.org/abs/2604.16826v1
• Date: Sat, 18 Apr 2026 04:33:16 GMT
• Title: Crowded in B-Space: Calibrating Shared Directions for LoRA Merging
• Abstract summary: Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance.<n>We show that the main source of LoRA merge interference comes from the output-side matrix $B$.<n>We propose Pico, a data-free method that calibrates $B$ before merge by downscaling over-shared directions and then rescaling the merged update.
• Score: 12.949322198287417
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance. Existing methods usually treat the LoRA update $ΔW = BA$ as a single object and do not distinguish the two LoRA matrices. We show that the main source of LoRA merge interference comes from the output-side matrix $B$. Across tasks, $B$ repeatedly uses a small set of shared directions, while $A$ remains much more task-specific. As a result, the merged adapter overemphasizes these shared directions, and task-specific information is lost. We propose Pico (Pre-merge interference calibration in output-space), a data-free method that calibrates $B$ before merge by downscaling over-shared directions and then rescaling the merged update. Pico plugs directly into existing merging methods such as Task Arithmetic, TIES, and TSV-M. Across eight different benchmarks from math, coding, finance, and medical domains, Pico improves average accuracy by 3.4-8.3 points over the corresponding base method and achieves the best overall average performance. Pico also enables merged adapters to outperform the LoRA trained with all task data. These results show that LoRA merging works better when the two LoRA matrices are treated separately.

Related papers

• LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates [48.87937677659571]
  Low-rank adaptation (LoRA) is the standard way to finetune large models.<n>We introduce LoRA-TSD, an adapter that treats every LoRA step as a tangent to the fixed-rank matrix manifold.<n>We give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero.
  arXiv  Detail & Related papers  (2026-09-02T15:41:45Z)
• Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach [58.97285859000893]
  We study whether $A$ should be shared across clients while $B$ remains client-specific (Share-A/Local-B) or whether $B$ should instead be shared while $A$ remains client-specific (Share-B/Local-A)<n>We propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance.
  arXiv  Detail & Related papers  (2026-08-10T15:38:45Z)
• Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter [31.066992554963093]
  We study the problem of merging $T$ LoRAs into a single rank-$r$ LoRA.<n>Existing-then-Compress pipelines treat the rank constraint as an afterthought.<n>We propose Compress-then-Merge (CtM), a reversed pipeline that enforces the rank-$r$ bottleneck before merging.
  arXiv  Detail & Related papers  (2026-06-02T14:42:46Z)
• Evolutionary Negative Module Pruning for Better LoRA Merging [46.397675691417675]
  Merging multiple Low-Rank Adaptation experts into a single backbone is a promising approach for efficient multi-task deployment.<n>Existing methods strive to alleviate interference via weight or subspace alignment.<n>We propose a plug-and-play LoRA pruning method to locate and exclude these modules prior to merging.
  arXiv  Detail & Related papers  (2026-04-20T03:13:18Z)
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs [7.13923757932177]
  Low-Rank Adaptation (LoRA) has become the leading Efficient Fine-Tuning (PEFT) method for Large Language Models (LLMs)<n>We introduce LoRAFusion, an efficient LoRA fine-tuning system for LLMs.<n>LoRAFusion achieves up to $1.96times$ ($1.47times$ on average) end-to-end speedup compared to Megatron-LM, and up to $1.46times$ ($1.29times$ on average) improvement over mLoRA.
  arXiv  Detail & Related papers  (2025-09-30T19:26:22Z)
• Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs [26.212332132619736]
  We propose an asymmetric multi-LoRA design with multiple $A$ matrices and a single shared $B$ in multi-task fine-tuning.<n>Our methods achieve more balanced performance across tasks with comparable or superior average accuracy relative to existing multi-LoRA approaches.
  arXiv  Detail & Related papers  (2025-09-29T19:16:14Z)
• Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update [50.36542772932594]
  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>There is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved.
  arXiv  Detail & Related papers  (2025-09-24T10:32:50Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• Selective Aggregation for Low-Rank Adaptation in Federated Learning [10.683530421910028]
  We introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices $A$ and $B$ to model the weight update.<n>We extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA.
  arXiv  Detail & Related papers  (2024-10-02T12:14:36Z)
• CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models [7.108651381160281]
  Low-Rank Adaptation (LoRA) strategy balances efficiency and performance in fine-tuning large models. We propose textbfCoRA: leveraging shared knowledge to optimize LoRA training by substituting its matrix $B$ with a common subspace from large models. Our experiments show that the first approach achieves the same efficacy as the original LoRA fine-tuning while being more efficient than halving parameters.
  arXiv  Detail & Related papers  (2024-08-31T12:48:27Z)
• ResLoRA: Identity Residual Mapping in Low-Rank Adaption [96.59370314485074]
  We propose ResLoRA, an improved framework of low-rank adaptation (LoRA) Our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-02-28T04:33:20Z)
• CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices [78.16679232748196]
  We introduce a Compression-Aware LoRA (CA-LoRA) framework to transfer Large Language Models (LLMs) to other tasks. Experiment results demonstrate that CA-LoRA outperforms the vanilla LoRA methods applied to a compressed LLM. The source code of CA-LoRA is available at https://github.com/thunlp/CA-LoRA.
  arXiv  Detail & Related papers  (2023-07-15T04:37:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.