Learning to Recover Task Experts from a Multi-Task Merged Model

• URL: http://arxiv.org/abs/2606.26902v1
• Date: Thu, 25 Jun 2026 11:34:33 GMT
• Title: Learning to Recover Task Experts from a Multi-Task Merged Model
• Abstract summary: Multi-task model merging aims to consolidate several task-specific experts into a unified model, yet static merging consistently suffers from parameter interference.<n>We show that such parameter perturbations can be modeled as affine transformation, which can be approximated as additive offsets.<n>We propose Recover Task eXpert (Re), a framework that predicts those offsets, in order to undo parameter interference and recover taskexpert performance from a single merged checkpoint.
• Score: 15.553883607402438
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-task model merging aims to consolidate several task-specific experts into a unified model, yet static merging consistently suffers from parameter interference. While dynamic merging models aim to bridge this gap, many works rely on the costly storage and loading of redundant expert components at inference. In this work, from the perspective of task expert, we view parameter interference as parameter perturbation introduced to each expert during merging process. We show that such parameter perturbations can be modeled as affine transformation, which can be approximated as additive offsets. Motivated by these, we propose Recover Task eXpert (ReTeX), a framework that predicts those offsets, in order to undo parameter interference and recover task-expert performance from a single merged checkpoint. To recover the appropriate expert when task identity is unknown, we introduce a router-free task identifier based on SVD subspace signatures computed offline before inference. At inference, the identifier selects the task whose subspace yields the smallest projection residual for a given input. As a result, ReTeX recovers over 95% of individual-expert performance in both vision and NLP domains, while significantly improving generalization to unseen tasks. Crucially, we also show that the parameter offset prediction leads to emergent adaptive interpolation of expert knowledge for out-of-distribution (OOD) tasks. ReTeX adaptively interpolates seen expert knowledge to handle unseen tasks. Our code is available at https://github.com/BAIKLAB/ReTeX

Related papers

• Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space [16.602503149462738]
  We formulate model merging from a new perspective of probabilistic inference under a product-of-experts (PoE) scenario.<n>We show that several existing model merging methods arise as special cases of our framework under energy designs.<n>We address this with a heavy-tailed PoE design based on Cauchy experts, which better captures the observed residual behavior while admitting a provably convergent inference procedure.
  arXiv  Detail & Related papers  (2026-07-02T04:30:51Z)
• PACT: Preserving Anchored Cores in Task-vectors for Model Merging [68.52455853496585]
  Model merging aims to combine multiple task-specific fine-tuned models into a single multi-task model.<n>Most existing model merging approaches follow the Task Arithmetic paradigm.<n>We propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their complements with the subspace of the pre-trained weights.
  arXiv  Detail & Related papers  (2026-06-17T02:48:35Z)
• TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning [33.07713704586859]
  We introduce TRACE, a novel approach for discovering Task-specific paRameters.<n>We conduct experiments across multiple standard benchmarks to demonstrate the superior performance of our proposed method.
  arXiv  Detail & Related papers  (2026-05-29T08:57:06Z)
• Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning [58.90602461025722]
  Multi-task instruct-tuning in large language models (LLMs) has been largely driven by multi-task instruct-tuning.<n>The training paradigm suffers from a key issue, named cross-task interference, due to conflicting gradients over shared parameters among different tasks.<n>We propose BADIT that decomposes LLM parameters into high-singular-value LoRA experts representing basic abilities.<n>We conduct extensive experiments on the SuperNI benchmark with 6 LLMs, and empirical results demonstrate that BADIT can outperform SOTA methods and mitigate the degree of cross-task interference.
  arXiv  Detail & Related papers  (2026-05-07T05:08:58Z)
• Model Merging in the Essential Subspace [78.5390284258307]
  Model merging aims to integrate multiple task-specific fine-tuned models into a single multi-task model without additional training.<n>Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models.<n>We propose ESM (Essential Subspace Merging), a robust framework for effective model merging.
  arXiv  Detail & Related papers  (2026-02-23T00:33:38Z)
• Subspace-Boosted Model Merging [18.15956635793621]
  Subspace Boosting raises merging efficacy for up to 20 expert models by large margins of more than 10%.<n>We propose employing Higher-Order Generalized Singular Value Decomposition to quantify task similarity.
  arXiv  Detail & Related papers  (2025-06-19T17:59:29Z)
• LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models [21.888139819188105]
  LLaVA-CMoE is a continual learning framework for large language models.<n> Probe-Guided Knowledge Extension mechanism determines when and where new experts should be added.<n>Probabilistic Task Locator assigns each task a dedicated, lightweight router.
  arXiv  Detail & Related papers  (2025-03-27T07:36:11Z)
• Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts [13.356826891549856]
  Multi-task model merging offers an efficient solution for integrating knowledge from multiple fine-tuned models.<n>Despite the promising performance of Task Arithmetic (TA), conflicts can arise among the task vectors.<n>We propose Task Arithmetic in Trust Region (TATR), which defines the trust region as dimensions in the model parameter space.
  arXiv  Detail & Related papers  (2025-01-25T04:09:56Z)
• Multi-Task Model Merging via Adaptive Weight Disentanglement [69.7292615212444]
  We introduce an Adaptive Weight Disentanglement method for model merging.<n>We successfully extract redundant vectors, and after their subtraction, the task vectors retain robust performance.
  arXiv  Detail & Related papers  (2024-11-27T20:08:55Z)
• Complexity Experts are Task-Discriminative Learners for Any Image Restoration [80.46313715427928]
  We introduce complexity experts" -- flexible expert blocks with varying computational complexity and receptive fields.<n>This preference effectively drives task-specific allocation, assigning tasks to experts with the appropriate complexity.<n>The proposed MoCE-IR model outperforms state-of-the-art methods, affirming its efficiency and practical applicability.
  arXiv  Detail & Related papers  (2024-11-27T15:58:07Z)
• Localizing Task Information for Improved Model Merging and Compression [61.16012721460561]
  We show that the information required to solve each task is still preserved after merging as different tasks mostly use non-overlapping sets of weights. We propose Consensus Merging, an algorithm that eliminates such weights and improves the general performance of existing model merging approaches.
  arXiv  Detail & Related papers  (2024-05-13T14:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.