Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout

• URL: http://arxiv.org/abs/2510.03262v2
• Date: Wed, 08 Oct 2025 00:05:16 GMT
• Title: Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout
• Abstract summary: Low-Rank Adaptation (LoRA), a popular fine-tuning method for large models, typically trains a module to represent a specific concept such as an object or a style.<n>We propose Orthogonal Monte Carlo Dropout, a mechanism that enforces strict orthogonality when combining sparse semantic vectors without extra time complexity.
• Score: 26.252083958846203
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose Orthogonal Monte Carlo Dropout, a mechanism that enforces strict orthogonality when combining sparse semantic vectors without extra time complexity. Low-Rank Adaptation (LoRA), a popular fine-tuning method for large models, typically trains a module to represent a specific concept such as an object or a style. When multiple LoRA modules are merged, for example to generate an object in a particular style, their outputs (semantic vectors) may interfere with each other. Our method guarantees that merged LoRA modules remain orthogonal and thus free from direct interference. However, empirical analysis reveals that such orthogonality does not lead to the semantic disentanglement highlighted in prior work on compositional adaptation. This finding suggests that inter-LoRA orthogonality alone may be insufficient for achieving true semantic compositionality, prompting a re-examination of its role in adapter merging.

Related papers

• Model-Agnostic FDR Control via Group Gaussian Mirror and Permutation SHAP [0.15293427903448018]
  We propose a grouped-feature FDR control framework for such settings.<n>For grouped linear models, we construct null-symmetric block-level mirror statistics with matrix-valued perturbations.<n>For sequential neural models, we combine Permutation SHAP derivatives as model-agnostic block-level importance scores with kernel-based dependence measure.
  arXiv  Detail & Related papers  (2026-08-02T04:39:27Z)
• PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry [0.0]
  Common hypothesis is that interference during adapter composition arises from overlap in linear parameter updates.<n>We test this hypothesis using DoRA-RBAC, a hierarchical adapter composition framework based on weight-decomposed low-rank adaptation.<n>Our results show that while single-domain performance matches LoRA, geometry-aware merging provides no consistent advantage over standard averaging in multi-domain settings.
  arXiv  Detail & Related papers  (2026-06-09T02:52:48Z)
• Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation [60.41721951405564]
  Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of segmentation foundation models for medical imaging.<n>Most federated LoRA methods adopt a uniform aggregation rule, which breaks under the encoder-decoder asymmetry in medical segmentation.<n>This mismatch entangles shared anatomy with site-specific biases and harms generalization.<n>We introduce a Subspace Orthogonality Regularizer that penalizes shared-local collinearity in the effective update space.
  arXiv  Detail & Related papers  (2026-06-07T15:39:28Z)
• Evolutionary Negative Module Pruning for Better LoRA Merging [46.397675691417675]
  Merging multiple Low-Rank Adaptation experts into a single backbone is a promising approach for efficient multi-task deployment.<n>Existing methods strive to alleviate interference via weight or subspace alignment.<n>We propose a plug-and-play LoRA pruning method to locate and exclude these modules prior to merging.
  arXiv  Detail & Related papers  (2026-04-20T03:13:18Z)
• JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models [78.56572636138429]
  We propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating.<n>We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
  arXiv  Detail & Related papers  (2026-04-17T15:38:37Z)
• Functionality-Oriented LLM Merging on the Fisher--Rao Manifold [14.349284217707575]
  Weight-space merging aims to combine multiple fine-tuned LLMs into a single model without retraining.<n>We derive a practical fixed-point algorithm using a lightweight spherical proxy that preserves norms and generalizes directly to multi-expert merging.
  arXiv  Detail & Related papers  (2026-03-05T09:08:38Z)
• Spanning the Visual Analogy Space with a Weight Basis of LoRAs [84.16188433935494]
  Visual analogy learning enables image manipulation through demonstration rather than textual description.<n>LoRWeB specializes the model for each analogy task at inference time through dynamic composition of learned transformation primitives.<n>We introduce two key components: (1) a learnable basis of LoRA modules, to span the space of different visual transformations, and (2) a lightweight encoder that dynamically selects and weighs these basis LoRAs.
  arXiv  Detail & Related papers  (2026-02-17T17:02:38Z)
• UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignment [30.711521764097657]
  We propose Preference-Modulated & Shared Low-Rank Adaptation (MoSLoRA) for ARM training.<n>MoSLoRA mitigates feature entanglement and enables precise control over preference trade-offs during inference.<n>We introduce the Unified Autoregressive Reward Model (UniARM), a novel framework for multi-objective test-time alignment.
  arXiv  Detail & Related papers  (2026-02-10T08:49:28Z)
• Identifying Intervenable and Interpretable Features via Orthogonality Regularization [48.938969291033665]
  We disentangle the decoder matrix into almost orthogonal features.<n>This reduces interference and superposition between the features, while keeping performance on the target dataset essentially unchanged.<n>Our code is available under $texttthttps://github.com/mrtzmllr/sae-icm$.
  arXiv  Detail & Related papers  (2026-02-04T16:29:14Z)
• QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation [52.024845354511555]
  We propose QR-LoRA, a novel fine-tuning framework leveraging QR decomposition for structured parameter updates.<n>Our key insight is that the Q matrix naturally minimizes interference between different visual features.<n>Experiments demonstrate that QR-LoRA achieves superior disentanglement in content-style fusion tasks.
  arXiv  Detail & Related papers  (2025-07-07T01:31:01Z)
• Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging [18.650279202312614]
  We propose a Decoupled and Orthogonal merging approach(DO-Merging)<n>By separating parameters into magnitude and direction components, we reduce the impact of magnitude differences on the directional alignment of the merged models.<n>We validate through extensive experiments across vision, language, and multi-modal domains that our proposed DO-Merging can achieve significantly higher performance than existing merging methods at a minimal cost.
  arXiv  Detail & Related papers  (2025-05-21T16:34:37Z)
• Closed-form merging of parameter-efficient modules for Federated Continual Learning [9.940242741914748]
  We introduce LoRM, an alternating optimization strategy that trains one LoRA matrix at a time.<n>We apply our proposed methodology to Federated Class-Incremental Learning (FCIL)<n>Our method demonstrates state-of-the-art performance across a range of FCIL scenarios.
  arXiv  Detail & Related papers  (2024-10-23T15:30:13Z)
• Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation [58.288682735160585]
  Low-Rank Adaptation (LoRA) is a popular technique for finetuning models. LoRA often under performs when compared to full- parameter fine-tuning. We present a framework that rigorously analyzes the adaptation rates of LoRA methods.
  arXiv  Detail & Related papers  (2024-10-10T18:51:53Z)
• LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks [52.46420522934253]
  We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks.<n>The method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble.
  arXiv  Detail & Related papers  (2024-05-23T11:10:32Z)
• Self-Supervised Learning Disentangled Group Representation as Feature [82.07737719232972]
  We show that existing Self-Supervised Learning (SSL) only disentangles simple augmentation features such as rotation and colorization. We propose an iterative SSL algorithm: Iterative Partition-based Invariant Risk Minimization (IP-IRM) We prove that IP-IRM converges to a fully disentangled representation and show its effectiveness on various benchmarks.
  arXiv  Detail & Related papers  (2021-10-28T16:12:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.