DC-Merge: Improving Model Merging with Directional Consistency

• URL: http://arxiv.org/abs/2603.06242v1
• Date: Fri, 06 Mar 2026 13:04:25 GMT
• Title: DC-Merge: Improving Model Merging with Directional Consistency
• Abstract summary: DC-Merge is a method for directional-consistent model merging.<n>It balances the energy distribution of each task vector by smoothing its singular values.<n>Experiments on vision and vision-language benchmarks show that DC-Merge consistently achieves state-of-the-art performance.
• Score: 62.02490833158024
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model merging aims to integrate multiple task-adapted models into a unified model that preserves the knowledge of each task. In this paper, we identify that the key to this knowledge retention lies in maintaining the directional consistency of singular spaces between merged multi-task vector and individual task vectors. However, this consistency is frequently compromised by two issues: i) an imbalanced energy distribution within task vectors, where a small fraction of singular values dominate the total energy, leading to the neglect of semantically important but weaker components upon merging, and ii) the geometric inconsistency of task vectors in parameter space, which causes direct merging to distort their underlying directional geometry. To address these challenges, we propose DC-Merge, a method for directional-consistent model merging. It first balances the energy distribution of each task vector by smoothing its singular values, ensuring all knowledge components are adequately represented. These energy-balanced vectors are then projected onto a shared orthogonal subspace to align their directional geometries with minimal reconstruction error. Finally, the aligned vectors are aggregated in the shared orthogonal subspace and projected back to the original parameter space. Extensive experiments on vision and vision-language benchmarks show that DC-Merge consistently achieves state-of-the-art performance in both full fine-tuning and LoRA settings. The implementation code is available at https://github.com/Tobeginwith/DC-Merge.

Related papers

• Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space [16.602503149462738]
  We formulate model merging from a new perspective of probabilistic inference under a product-of-experts (PoE) scenario.<n>We show that several existing model merging methods arise as special cases of our framework under energy designs.<n>We address this with a heavy-tailed PoE design based on Cauchy experts, which better captures the observed residual behavior while admitting a provably convergent inference procedure.
  arXiv  Detail & Related papers  (2026-07-02T04:30:51Z)
• Essential Subspace Merging for Multi-Task Learning [78.5390284258307]
  We analyze the output shifts induced by task updates and observe that their energy is concentrated in a small number of principal directions.<n>Most remaining directions carry little task-relevant energy, but their accumulation across multiple task updates can cause severe interference during merging.<n>Based on ESD, we introduce Essential Subspace Merging (ESM), a training-free static merging method.<n>We extend ESM to ESM++, a training-free dynamic merging method that decomposes task-specific residuals into low-rank experts.
  arXiv  Detail & Related papers  (2026-06-17T15:07:11Z)
• PACT: Preserving Anchored Cores in Task-vectors for Model Merging [68.52455853496585]
  Model merging aims to combine multiple task-specific fine-tuned models into a single multi-task model.<n>Most existing model merging approaches follow the Task Arithmetic paradigm.<n>We propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their complements with the subspace of the pre-trained weights.
  arXiv  Detail & Related papers  (2026-06-17T02:48:35Z)
• Model Merging: Foundations and Algorithms [4.528573838858818]
  This thesis studies model merging, combining independently trained neural networks directly in weight space.<n>In the single-task setting, we introduce C$2$M$3$, a cycle-consistent merging algorithm based on Frank-Wolfe optimization.<n>In the multi-task setting, we first develop a theoretical account of task vectors as approximate gradients.<n>We then present MASS, an input-adaptive routing method that uses TSV geometry to select task-relevant subspaces at inference time.
  arXiv  Detail & Related papers  (2026-05-02T19:06:35Z)
• Functionality-Oriented LLM Merging on the Fisher--Rao Manifold [14.349284217707575]
  Weight-space merging aims to combine multiple fine-tuned LLMs into a single model without retraining.<n>We derive a practical fixed-point algorithm using a lightweight spherical proxy that preserves norms and generalizes directly to multi-expert merging.
  arXiv  Detail & Related papers  (2026-03-05T09:08:38Z)
• Model Merging in the Essential Subspace [78.5390284258307]
  Model merging aims to integrate multiple task-specific fine-tuned models into a single multi-task model without additional training.<n>Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models.<n>We propose ESM (Essential Subspace Merging), a robust framework for effective model merging.
  arXiv  Detail & Related papers  (2026-02-23T00:33:38Z)
• Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport [29.006391770977796]
  OTMF is a novel model merging framework rooted in optimal transport theory.<n>It supports a continual fusion paradigm that incrementally integrates each new task vector without revisiting previous ones.<n>We conduct comprehensive experiments on multiple vision and language benchmarks, and results show that OTMF achieves state-of-the-art performance in terms of both accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-11-24T15:27:47Z)
• Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging [83.5273168208788]
  Model merging aims to integrate task-specific abilities from individually fine-tuned models into a single model without extra training.<n>The merged model often suffers from notable performance degradation due to the conflicts caused by task-irrelevant redundancy in task vectors.<n>We propose Purifying TAsk Vectors (PAVE) in knowledge-aware subspace to overcome these challenges.
  arXiv  Detail & Related papers  (2025-10-16T14:02:57Z)
• Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking [17.095655627061934]
  We present Decom-Renorm-Merge (DRM), a simple yet effective approach that leverages Singular Value Decomposition to decompose and coordinate weight matrices into an aligned joint space.<n>Our experimental results show that DRM outperforms several state-of-the-art merging techniques across full finetuning and low-rank adaptation settings.
  arXiv  Detail & Related papers  (2025-05-29T05:37:53Z)
• Multi-Task Model Merging via Adaptive Weight Disentanglement [69.7292615212444]
  We introduce an Adaptive Weight Disentanglement method for model merging.<n>We successfully extract redundant vectors, and after their subtraction, the task vectors retain robust performance.
  arXiv  Detail & Related papers  (2024-11-27T20:08:55Z)
• ComPtr: Towards Diverse Bi-source Dense Prediction Tasks via A Simple yet General Complementary Transformer [71.82644727907146]
  We propose a novel $underlineComP$lementary $underlinetr$ansformer, $textbfComPtr$, for diverse bi-source dense prediction tasks.<n>ComPtr treats different inputs equally and builds an efficient dense interaction model in the form of sequence-to-sequence on top of the transformer.
  arXiv  Detail & Related papers  (2023-07-23T15:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.