Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging

• URL: http://arxiv.org/abs/2606.18521v1
• Date: Tue, 16 Jun 2026 22:22:40 GMT
• Title: Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging
• Abstract summary: Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT)<n>Recent studies reveal that RLVR induces highly sparse and off-principal parameter updates compared to SFT.<n>We propose Sensitivity-aware Resolving Merging (SAR-Merging), a merging recipe tailored for the unique structure of RLVR parameter spaces.
• Score: 35.66009296523317
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT) in eliciting reasoning intelligence and resisting catastrophic forgetting. Recent studies further reveal that RLVR induces highly sparse and off-principal parameter updates compared to SFT. This naturally raises the question: does such sparsity make RLVR models more amenable to model merging? If so, model merging would offer a scalable, training-free path to aggregate diverse reasoning capabilities from independently trained RLVR models. Surprisingly, we find the opposite, uncovering a sparsity curse: the sparse RLVR updates are spread farther apart in parameter space, forming near-orthogonal shortcuts that make aggregation inherently fragile. This is likely rooted in the stochasticity of RL optimization and the diversity of emergent reasoning patterns. Unlike SFT models that converge to shared, flat basins and merge naturally, RLVR models suffer severe degradation under standard merging methods. Through systematic empirical analysis of the update geometry, we characterize the mechanisms behind this failure and propose Sensitivity-aware Resolving Merging (SAR-Merging), a merging recipe tailored for the unique structure of RLVR parameter spaces. SAR-Merging resolves conflicts in overlapping update regions via Fisher Information-based sensitivity arbitration, followed by magnitude-aware sparsification and rescaling to preserve fragile reasoning pathways. Experiments on mathematical and coding benchmarks demonstrate that SAR-Merging substantially outperforms existing merging methods on RLVR models, enabling both single-task enhancement and multi-capability fusion.

Related papers

• Boosting LLM Exploration via Weak-Model Guidance in RLVR [36.33733929880283]
  We propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR.<n>Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models.<n>Our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.
  arXiv  Detail & Related papers  (2026-08-27T17:45:50Z)
• BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences? [5.571635164072802]
  Reinforcement learning with verifiable rewards (RLVR) has improved the performance of large language models (LLMs) across a variety of reasoning tasks.<n>We investigate the nature of test-time exploration in RLVR-trained LLMs by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces.<n>Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic, and is accompanied by a significant reduction in semantic branching entropy.
  arXiv  Detail & Related papers  (2026-08-03T20:37:54Z)
• Understanding Diversity Collapse in RLVR via the Lens of Overtraining [78.37408098404312]
  Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models.<n>We formalize this diversity collapse through the lens of emphovertraining<n>We propose emphBayesian Boundary Gating (BBG) which redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary.
  arXiv  Detail & Related papers  (2026-06-13T20:13:37Z)
• Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors [26.233592394784868]
  We analyze SFT and RLVR through the lens of task vectors.<n>We propose Decoupled Test-time Synthesis (DoTS) to allow SFT and RLVR checkpoints to be trained independently.
  arXiv  Detail & Related papers  (2026-05-01T12:20:44Z)
• Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration [88.7332439088792]
  scaling reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs) has emerged as an effective training paradigm.<n>We propose the textbfNonlinear textbfExtrapolation of low-rank trajectories (textbfNExt)<n>Our method reduces computational overhead by approximately 37.5% while remaining compatible with a wide range of RLVR algorithms and tasks.
  arXiv  Detail & Related papers  (2026-04-13T13:28:12Z)
• MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation [24.618644100413018]
  Reinforcement learning (RL) has been successfully applied to autoregressive (AR) and diffusion models.<n> extending RL to hybrid AR-diffusion frameworks remains challenging due to interleaved inference and noisy log-probability estimation.<n>In this work, we study masked autoregressive models (MAR) and show that the diffusion head plays a critical role in training dynamics.
  arXiv  Detail & Related papers  (2026-04-08T11:30:35Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging [20.429700094073684]
  We propose Sparse Complementary Fusion with reverse KL (SCF-RKL), a novel model merging framework that explicitly controls functional interference through sparse, distribution-aware updates.<n>We evaluate SCF-RKL across a wide range of model scales and architectures, covering both reasoning-focused and instruction-tuned models.
  arXiv  Detail & Related papers  (2026-02-12T08:45:42Z)
• Do Reasoning Models Enhance Embedding Models? [48.43242995118735]
  State-of-the-art embedding models are increasingly derived from decoder-only Large Language Model backbones adapted via contrastive learning.<n>We show that embedding models from RLVR-tuned backbones yield no consistent performance advantage over their base counterparts when subjected to identical training recipes.
  arXiv  Detail & Related papers  (2026-01-29T02:48:34Z)
• The Path Not Taken: RLVR Provably Learns Off the Principals [85.41043469428365]
  We show that sparsity is a surface artifact of a model-conditioned optimization bias.<n>We mechanistically explain these dynamics with a Three-Gate Theory.<n>We provide a parameter-level characterization of RLVR's learning dynamics.
  arXiv  Detail & Related papers  (2025-11-11T18:49:45Z)
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward [57.56453588632619]
  A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance.<n>This is often accompanied by catastrophic forgetting, where models lose previously acquired skills.<n>We argue that standard RLVR objectives lack a crucial mechanism for knowledge retention.
  arXiv  Detail & Related papers  (2025-09-09T06:34:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.