Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy

• URL: http://arxiv.org/abs/2603.26299v1
• Date: Fri, 27 Mar 2026 11:07:46 GMT
• Title: Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy
• Abstract summary: Merging multiple Low-Rank Adaptation (LoRA) modules is promising for constructing general-purpose systems.<n>We propose TARA-Merging, which aligns merging weights using a preference-weighted cross-entropy pseudo-loss.<n>Across eight vision and six NLI benchmarks, TARA-Merging consistently outperforms vanilla and LoRA-aware baselines.
• Score: 50.63908869296697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Merging multiple Low-Rank Adaptation (LoRA) modules is promising for constructing general-purpose systems, yet challenging because LoRA update directions span different subspaces and contribute unevenly. When merged naively, such mismatches can weaken the directions most critical to certain task losses while overemphasizing relatively less important ones, ultimately reducing the model's ability to represent all tasks faithfully. We revisit this problem through two perspectives: subspace coverage, which captures how broadly LoRA directions cover diverse representational directions, and anisotropy, which reflects the imbalance of influence across those directions. We propose TARA-Merging (Task-Rank Anisotropy Alignment), which aligns merging weights using a preference-weighted cross-entropy pseudo-loss while preserving task-relevant LoRA subspaces. This ensures broad subspace coverage and mitigates anisotropy via direction-wise reweighting. Across eight vision and six NLI benchmarks, TARA-Merging consistently outperforms vanilla and LoRA-aware baselines, demonstrating strong robustness and generalization, and highlighting the importance of addressing both subspace coverage and anisotropy in LoRA merging.

Related papers

• Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation [54.74045834035952]
  Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs.<n>We propose a rank-aware hyperbolic alignment (RAHA) that combines hierarchical geometry with explicit alignment-capacity control.<n>RAHA demonstrates competitive cross-modal retrieval and improved transfer indicators under fixed budgets.
  arXiv  Detail & Related papers  (2026-06-28T15:41:31Z)
• SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling [17.873016968298483]
  Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.<n>We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that re parameterizes a rank-rtot update as a sum of K static experts.
  arXiv  Detail & Related papers  (2026-06-27T03:25:36Z)
• Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models [73.73784737382178]
  We introduce a robust approach that generates synthetic datasets with controlled variations, fine-tunes a LoRA adapter on each dataset, and extracts a LoRA sub-space associated with each type of variation.<n>We show that such a reduced LoRA subspace, despite being derived entirely from synthetic data, generalizes to real datasets.
  arXiv  Detail & Related papers  (2026-04-11T08:33:13Z)
• Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA [50.97792275353563]
  We introduce a novel framework that restructures a single Low-Rank Adaptation (LoRA) module as a decomposable Rank-1 Expert Pool.<n>Our method learns to dynamically compose a sparse, task-specific update by selecting from this expert pool, guided by the semantics of the [Guided] token.
  arXiv  Detail & Related papers  (2026-01-30T10:54:51Z)
• Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces [12.630494786258842]
  Low-Rank Adaptation (LoRA) enables efficient Continual Learning but often suffers from catastrophic forgetting.<n>We propose PS-LoRA, a framework designed to resolve conflicts by aligning updates within the optimization subspace.<n>Our approach employs a dual-regularization objective that penalizes conflicting directions and constrains magnitude deviations to ensure consistency with prior knowledge.
  arXiv  Detail & Related papers  (2025-11-28T15:34:36Z)
• NP-LoRA: Null Space Projection Unifies Subject and Style in LoRA Fusion [16.64405108290577]
  Low-Rank Adaptation (LoRA) fusion is a key technique for reusing and composing learned subject and style representations.<n>Existing methods rely on weight-based merging, where one LoRA often dominates the other.<n>We propose Null Space Projection LoRA, a projection-based framework for LoRA fusion that enforces subspace separation.
  arXiv  Detail & Related papers  (2025-11-14T08:06:01Z)
• Latent Space Factorization in LoRA [10.994747174370099]
  Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning.<n>We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA) to learn two distinct latent spaces.<n>Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information.
  arXiv  Detail & Related papers  (2025-10-22T14:37:20Z)
• Two Is Better Than One: Rotations Scale LoRAs [26.617019830475172]
  Low-Rank Adaptation (LoRA)-based Mixture-of-Experts (MoE) facilitates large language models (LLMs) to efficiently adapt to diverse tasks.<n>Traditional gating mechanisms that route inputs to the best experts may fundamentally hinder LLMs' scalability.<n>We propose RadarGate, a novel geometrically inspired gating method that introduces rotational operations of LoRAs representations.
  arXiv  Detail & Related papers  (2025-05-29T07:22:43Z)
• SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization [2.594346658179846]
  Low-Rank Adaptation (LoRA) constrains updates to a fixed low-rank subspace.<n>We introduce Subspace Recomposition in Low-Rank Adaptation (SRLoRA) via importance-based fusion and reinitialization.<n> SRLoRA consistently achieves faster convergence and improved accuracy over standard LoRA.
  arXiv  Detail & Related papers  (2025-05-18T14:12:40Z)
• BeamLoRA: Beam-Constraint Low-Rank Adaptation [51.52097743781401]
  Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods.<n>We propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution.
  arXiv  Detail & Related papers  (2025-02-19T10:33:22Z)
• LoRA vs Full Fine-tuning: An Illusion of Equivalence [73.5303340531806]
  We study how Low-Rank Adaptation (LoRA) and full-finetuning change pre-trained models.<n>We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure.<n>We extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized to the intruder dimension.
  arXiv  Detail & Related papers  (2024-10-28T17:14:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.