TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging

• URL: http://arxiv.org/abs/2606.05016v1
• Date: Wed, 03 Jun 2026 15:39:37 GMT
• Title: TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging
• Abstract summary: We argue that task and domain adapters exhibit a consistent depth-dependent asymmetry across transformer architectures.<n>We propose a training-free algorithm that exploits this structure through calibrated probe-guided per-layer gating and per-component subspace-aware merging.<n>$textbfTaDA$ produces a standard rank-$r$ LoRA adapter with zero inference overhead.
• Score: 6.356259006504384
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Combining a task LoRA adapter with a domain LoRA adapter into a single unified model is a practical yet largely unexplored challenge. Existing methods treat both adapters as symmetric peers, applying uniform weights across all layers. We argue that task and domain adapters exhibit a consistent depth-dependent asymmetry across transformer architectures. Domain dominance increases with layer depth, while shallower layers retain stronger task-relevant signals. Motivated by this observation, we propose $\textbf{TaDA}$ ($\textbf{Ta}$sk-$\textbf{D}$omain LoR$\textbf{A}$ Merging), a training-free algorithm that exploits this structure through calibrated probe-guided per-layer gating and per-component subspace-aware merging. The gating assigns individual weights per layer and projection type using a probe signal proved invariant to adapter weight magnitude. The merging discards conflicting singular directions before combining the remaining components. $\textbf{TaDA}$ produces a standard rank-$r$ LoRA adapter with zero inference overhead. On six scientific QA benchmarks with Llama-2-7B, TaDA achieves an average accuracy of 0.452, outperforming DARE-TIES by +3.6 percentage points and obtaining the best result on all six benchmarks. On six image classification benchmarks with ViT-L/16, TaDA reaches 85.9\% average accuracy, improving over the strongest merging baseline while leading in three of the six individual benchmarks.

Related papers

• LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates [48.87937677659571]
  Low-rank adaptation (LoRA) is the standard way to finetune large models.<n>We introduce LoRA-TSD, an adapter that treats every LoRA step as a tangent to the fixed-rank matrix manifold.<n>We give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero.
  arXiv  Detail & Related papers  (2026-09-02T15:41:45Z)
• The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning [3.860898721527665]
  LoRA fine-tuning can create intruder dimensions that drive catastrophic forgetting.<n>No theory has predicted, layer by layer on measured spectra, when they appear.<n>Full fine-tuning disperses its update far below the threshold of every layer.<n>Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting.
  arXiv  Detail & Related papers  (2026-07-26T15:17:41Z)
• TALAN: Task-Aligned Latent Adaptation Networks for Targeted Post-Training of Large Language Models [8.47008876232157]
  We introduce TALAN (Task-Aligned Latent Adaptation Networks), a sequence-conditioned latent side path inserted into a transformer's residual stream.<n> TALAN compresses the active sequence into latent memory, remixes it into token-level perturbations, and writes them back through a controlled residual update.<n>Across four Qwen3-family backbones and four STEM/code benchmarks, TALAN improves matched LoRA and DoRA baselines.
  arXiv  Detail & Related papers  (2026-06-05T04:34:50Z)
• From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging [26.22748334252526]
  Training strong large language models (LLMs) requires high-quality supervision, which is often scarce.<n>Recent work shows that paired preference data from weak-weaker model pairs can provide an effective supervision signal through relative quality deltas.<n>This motivates a key research question: can multiple "weak" signals be constructively aggregated for improving strong models (e.g., Qwen3 8B)?<n>PDA is the first framework that derives a preference delta from each weak-weaker model pair, instantiates it as a LoRA adapter learned through preference optimization, and aggregates the resulting deltas via LoRA merging.
  arXiv  Detail & Related papers  (2026-05-29T21:00:29Z)
• Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation [0.0]
  We study quality-latency-resource trade-offs in a retrieval-augmented generation (RAG) system that uses Low-Rank Adaptation (LoRA) of the generator.<n>We build a benchmark of 5,144 question-answer pairs over the official documentation and combine it with a fixed hybrid-retrieval pipeline (BGE-M3 dense, BGE-M3 native, Reciprocal Rank Fusion, cross-encoder reranking)
  arXiv  Detail & Related papers  (2026-05-27T09:37:55Z)
• FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation [0.0]
  Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix.<n>We compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally.<n>The resulting adapter is a standard LoRA with a per-layer rank pattern.
  arXiv  Detail & Related papers  (2026-05-16T04:03:21Z)
• Crowded in B-Space: Calibrating Shared Directions for LoRA Merging [12.949322198287417]
  Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance.<n>We show that the main source of LoRA merge interference comes from the output-side matrix $B$.<n>We propose Pico, a data-free method that calibrates $B$ before merge by downscaling over-shared directions and then rescaling the merged update.
  arXiv  Detail & Related papers  (2026-04-18T04:33:16Z)
• Sensitivity-Positional Co-Localization in GQA Transformers [0.0]
  We investigate a fundamental structural question in Grouped Query Attention transformers.<n>Do the layers most sensitive to task correctness coincide with the layers where positional encoding adaptation has the greatest leverage?<n>We introduce LSLORA, which restricts LoRA adaptation to layers identified via a novel correctness-differential hidden-state metric, and GARFA, which attaches 8 learnable per-KV-head scalar multipliers to each targeted layer.
  arXiv  Detail & Related papers  (2026-04-09T03:41:33Z)
• WeightLoRA: Keep Only Necessary Adapters [76.32368157312477]
  Low-rank adaptation ($texttLoRA$) adds trainable adapters to selected layers.<n>We propose a novel method, $textttWeightLoRA$, which overcomes this issue by adaptive selection of the most critical $textttLoRA$ heads.<n>We conduct experiments for a series of competitive benchmarks and DeBERTa, BART, and Llama models, comparing our method with different adaptive approaches.
  arXiv  Detail & Related papers  (2025-06-03T10:33:16Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• Mitigating the Alignment Tax of RLHF [76.4300447532456]
  aligning LLMs under Reinforcement Learning with Human Feedback can lead to forgetting pretrained abilities, also known as the alignment tax. We propose model averaging to maximize alignment performance while incurring minimal alignment tax. We validate HMA's performance across a range of RLHF algorithms over OpenLLaMA-3B and further extend our findings to Mistral-7B.
  arXiv  Detail & Related papers  (2023-09-12T14:16:54Z)
• Multi-Head Adapter Routing for Cross-Task Generalization [56.75667096355806]
  Polytropon learns an inventory of adapters and a routing function that selects a subset of adapters for each task during both pre-training and few-shot adaptation. We find that routing is most beneficial during multi-task pre-training rather than during few-shot adaptation.
  arXiv  Detail & Related papers  (2022-11-07T19:35:55Z)
• Semantic-Aligned Matching for Enhanced DETR Convergence and Multi-Scale Feature Fusion [95.7732308775325]
  The proposed DEtection TRansformer (DETR) has established a fully end-to-end paradigm for object detection. DETR suffers from slow training convergence, which hinders its applicability to various detection tasks. We design Semantic-Aligned-Matching DETR++ to accelerate DETR's convergence and improve detection performance.
  arXiv  Detail & Related papers  (2022-07-28T15:34:29Z)
• Disentangle Your Dense Object Detector [82.22771433419727]
  Deep learning-based dense object detectors have achieved great success in the past few years and have been applied to numerous multimedia applications such as video understanding. However, the current training pipeline for dense detectors is compromised to lots of conjunctions that may not hold. We propose Disentangled Dense Object Detector (DDOD), in which simple and effective disentanglement mechanisms are designed and integrated into the current state-of-the-art detectors.
  arXiv  Detail & Related papers  (2021-07-07T00:52:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.