When Model Merging Rivals Joint Multi-Task Reinforcement Learning: A Task-Vector Geometry Analysis

• URL: http://arxiv.org/abs/2607.16062v1
• Date: Fri, 17 Jul 2026 15:41:09 GMT
• Title: When Model Merging Rivals Joint Multi-Task Reinforcement Learning: A Task-Vector Geometry Analysis
• Abstract summary: Training difficulty-1 and difficulty-2 Qwen3-8B specialists on the AppWorld agent benchmark with LOOP.<n>We merge them (TIES, RAM+) and pit the result against a jointly trained model on the same data.<n>On task-goal completion, merging matches joint RL -- and every merge variant is statistically indistinguishable.
• Score: 0.16921396880325779
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model merging is promoted as a substitute for joint multi-task training, yet in the reinforcement-learning setting this substitution is essentially never tested against the baseline it claims to replace: methods merge independently released agents precisely because a joint model is unavailable. We build the missing comparison. Training difficulty-1 and difficulty-2 Qwen3-8B specialists on the AppWorld agent benchmark with LOOP, we merge them (TIES, RAM+) and pit the result against a jointly trained model on the same data. On task-goal completion, merging matches joint RL -- and every merge variant is statistically indistinguishable. To explain why merge method does not matter here, we measure the geometry of the specialists' task vectors, which carries no task-sampling noise: they are near-orthogonal (cosine 0.06 - 0.10) despite ~65% support overlap, a small, shared direction that grows over training and that we calibrate against a random-init floor and a same-run ceiling to confirm it reflects learning, not the low-rank parameterization. Because direction and support are decoupled, support and sign-based merging (RAM, TIES) collapse to near-uniform averaging. We release all code and statistics.

Related papers

• Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognition [4.800831341795478]
  We use two Gemma-3-1B-IT finetunes on two complementary safety objectives: CARES harm-level classification and WildJailbreak adversarial refusal.<n>We merge the two fine-tunes using Linear, SLERP, TIES, and DARE-TIES, and evaluate the merged models on classification accuracy, attack resistance, and benign compliance.
  arXiv  Detail & Related papers  (2026-07-26T19:10:34Z)
• PACT: Preserving Anchored Cores in Task-vectors for Model Merging [68.52455853496585]
  Model merging aims to combine multiple task-specific fine-tuned models into a single multi-task model.<n>Most existing model merging approaches follow the Task Arithmetic paradigm.<n>We propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their complements with the subspace of the pre-trained weights.
  arXiv  Detail & Related papers  (2026-06-17T02:48:35Z)
• The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning [0.5854803320592717]
  We prove optimality in a linear-Gaussian model (Thm. A), necessity of range coverage for any quadratic penalty that zeros deployment drift (Thm. G)<n>Thirteen blocks (ML through Qwen2.5-7B) test matched vs isotropic vs wrong-direction penalties on geometry and deployment drift.<n>We claim a falsifiable design recipe: estimate Sigma_task, match Sigma', run the controls, report task and geometry separately.
  arXiv  Detail & Related papers  (2026-05-21T17:53:28Z)
• Label-Free Cross-Task LoRA Merging with Null-Space Compression [50.63908869296697]
  We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry.<n>NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks.
  arXiv  Detail & Related papers  (2026-03-27T11:34:41Z)
• An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse [19.740211950583525]
  We identify and characterize the phenomenon of task-level merging collapse.<n>We show that representational incompatibility between tasks is strongly correlated with merging collapse.<n>We provide a theoretical explanation on this phenomenon through rate-distortion theory with a dimension-dependent bound.
  arXiv  Detail & Related papers  (2026-03-10T10:18:32Z)
• DC-Merge: Improving Model Merging with Directional Consistency [62.02490833158024]
  DC-Merge is a method for directional-consistent model merging.<n>It balances the energy distribution of each task vector by smoothing its singular values.<n>Experiments on vision and vision-language benchmarks show that DC-Merge consistently achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-06T13:04:25Z)
• Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching [66.39914384073145]
  We propose a self-consistency framework that turns cheap diffusion-sampled reasoning into a reusable pool of step-level candidates.<n>We find that step-level recombination is most beneficial on harder problems.<n>Our training-free framework improves average accuracy by up to 2 across six math and coding tasks.
  arXiv  Detail & Related papers  (2026-02-26T11:08:39Z)
• Behavior Knowledge Merge in Reinforced Agentic Models [48.89546963456286]
  Reinforcement learning is central to post-training, particularly for agentic models that require specialized reasoning behaviors.<n>Existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models.<n>We propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models.
  arXiv  Detail & Related papers  (2026-01-20T03:56:53Z)
• Model Merging with Functional Dual Anchors [21.76214716818033]
  Model merging is an efficient strategy for integrating knowledge from multiple finetuned checkpoints of a shared foundation model.<n>We propose Functional Dual Anchors (FDAs), a framework that instead models the input-representation space.<n>FDAs are synthetic inputs whose induced gradients align with task vectors, capturing task-specific functional shifts relative to the pretrained model.
  arXiv  Detail & Related papers  (2025-10-24T07:54:06Z)
• An Empirical Study of Multimodal Model Merging [148.48412442848795]
  Model merging is a technique that fuses multiple models trained on different tasks to generate a multi-task solution. We conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture. We propose two metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes.
  arXiv  Detail & Related papers  (2023-04-28T15:43:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.