The Path Not Taken: RLVR Provably Learns Off the Principals

• URL: http://arxiv.org/abs/2511.08567v1
• Date: Wed, 12 Nov 2025 02:05:16 GMT
• Title: The Path Not Taken: RLVR Provably Learns Off the Principals
• Abstract summary: We show that sparsity is a surface artifact of a model-conditioned optimization bias.<n>We mechanistically explain these dynamics with a Three-Gate Theory.<n>We provide a parameter-level characterization of RLVR's learning dynamics.
• Score: 85.41043469428365
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) reliably improves the reasoning performance of large language models, yet it appears to modify only a small fraction of parameters. We revisit this paradox and show that sparsity is a surface artifact of a model-conditioned optimization bias: for a fixed pretrained model, updates consistently localize to preferred parameter regions, highly consistent across runs and largely invariant to datasets and RL recipes. We mechanistically explain these dynamics with a Three-Gate Theory: Gate I (KL Anchor) imposes a KL-constrained update; Gate II (Model Geometry) steers the step off principal directions into low-curvature, spectrum-preserving subspaces; and Gate III (Precision) hides micro-updates in non-preferred regions, making the off-principal bias appear as sparsity. We then validate this theory and, for the first time, provide a parameter-level characterization of RLVR's learning dynamics: RLVR learns off principal directions in weight space, achieving gains via minimal spectral drift, reduced principal-subspace rotation, and off-principal update alignment. In contrast, SFT targets principal weights, distorts the spectrum, and even lags RLVR. Together, these results provide the first parameter-space account of RLVR's training dynamics, revealing clear regularities in how parameters evolve. Crucially, we show that RL operates in a distinct optimization regime from SFT, so directly adapting SFT-era parameter-efficient fine-tuning (PEFT) methods can be flawed, as evidenced by our case studies on advanced sparse fine-tuning and LoRA variants. We hope this work charts a path toward a white-box understanding of RLVR and the design of geometry-aware, RLVR-native learning algorithms, rather than repurposed SFT-era heuristics.

Related papers

• Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR [22.761873383120786]
  Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning paradigm.<n>Two structurally LoRA variants, PiSSA and MiLoRA, can outperform standard LoRA under Reinforcement learning with verifiable rewards (RLVR) and may even exhibit training instability.
  arXiv  Detail & Related papers  (2026-06-30T15:27:54Z)
• Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging [35.66009296523317]
  Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT)<n>Recent studies reveal that RLVR induces highly sparse and off-principal parameter updates compared to SFT.<n>We propose Sensitivity-aware Resolving Merging (SAR-Merging), a merging recipe tailored for the unique structure of RLVR parameter spaces.
  arXiv  Detail & Related papers  (2026-06-16T22:22:40Z)
• On the Geometry of On-Policy Distillation [22.873898953554605]
  We study on-policy distillation (OPD) to improve large language model reasoning.<n>We compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR)<n>We find that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.
  arXiv  Detail & Related papers  (2026-06-05T09:20:15Z)
• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR [51.935533482549545]
  We show that RLVR may exhibit implicit reward overfitting to the training dataset.<n>The model can achieve satisfactory performance on the test set even when its rewards remain relatively low during the training process.
  arXiv  Detail & Related papers  (2026-05-07T16:30:28Z)
• Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors [26.233592394784868]
  We analyze SFT and RLVR through the lens of task vectors.<n>We propose Decoupled Test-time Synthesis (DoTS) to allow SFT and RLVR checkpoints to be trained independently.
  arXiv  Detail & Related papers  (2026-05-01T12:20:44Z)
• Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration [88.7332439088792]
  scaling reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs) has emerged as an effective training paradigm.<n>We propose the textbfNonlinear textbfExtrapolation of low-rank trajectories (textbfNExt)<n>Our method reduces computational overhead by approximately 37.5% while remaining compatible with a wide range of RLVR algorithms and tasks.
  arXiv  Detail & Related papers  (2026-04-13T13:28:12Z)
• Curvature-Guided LoRA: Steering in the pretrained NTK subspace [60.35296431630704]
  We introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs.<n>We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient.<n>Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information.
  arXiv  Detail & Related papers  (2026-03-31T14:46:39Z)
• GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR [10.820638016337869]
  We propose GeoRA, which exploits the anisotropic and compressible nature of RL update subspaces.<n>GeoRA mitigates optimization bottlenecks caused by geometric misalignment.<n>It consistently outperforms established low-rank baselines on key mathematical benchmarks.
  arXiv  Detail & Related papers  (2026-01-14T10:41:34Z)
• High-Rank Structured Modulation for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>We present textbfSMoA, a high-rank textbfStructured textbfMOdulation textbfAdapter that uses fewer trainable parameters while maintaining a higher rank.
  arXiv  Detail & Related papers  (2026-01-12T13:06:17Z)
• Evaluating Parameter Efficient Methods for RLVR [38.45552186628944]
  Reinforcement Learning with Verifiable Rewards (RLVR) incentivizes language models to enhance their reasoning capabilities through verifiable feedback.<n>While methods like LoRA are commonly used, the optimal PEFT architecture for RLVR remains unidentified.<n>We conduct the first comprehensive evaluation of over 12 PEFT methodologies across the DeepSeek-R1-Distill families on mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-12-29T03:13:08Z)
• Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning [93.19037653970622]
  We introduce Spatial-SSRL, a self-supervised RL paradigm that derives verifiable signals directly from ordinary RGB or RGB-D images.<n>Training on our tasks substantially improves spatial reasoning while preserving general visual capabilities.<n>Our results show that simple, intrinsic supervision enables RLVR at scale and provides a practical route to stronger spatial intelligence in LVLMs.
  arXiv  Detail & Related papers  (2025-10-31T16:30:08Z)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models [20.320268628019047]
  This work identifies two fundamental properties of RL-induced parameter updates in large language models.<n>We propose AlphaRL, a plug-in acceleration framework that extrapolates the final parameter update using a short early training window.
  arXiv  Detail & Related papers  (2025-10-01T06:13:50Z)
• The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features [1.7832672957068079]
  We introduce Feature Steering with Reinforcement Learning, a framework that trains a lightweight adapter to steer model behavior by modulating interpretable sparse features.<n>We show that this mechanism is principled and expressive enough to approximate the behavioral shifts of post-training processes.<n>Overall, FSRL offers an interpretable control interface and a practical way to diagnose how preference optimization pressures manifest at the feature level.
  arXiv  Detail & Related papers  (2025-09-16T10:32:40Z)
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward [57.56453588632619]
  A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance.<n>This is often accompanied by catastrophic forgetting, where models lose previously acquired skills.<n>We argue that standard RLVR objectives lack a crucial mechanism for knowledge retention.
  arXiv  Detail & Related papers  (2025-09-09T06:34:32Z)
• Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models [0.0]
  It is often assumed thatReinforcement learning (RL) fine-tuning requires updating most of a model's parameters.<n>We call this phenomenon RL-induced parameter update sparsity.<n>We show that fine-tuning only this sparse subnetwork recovers full model performance and yields parameters nearly identical to the fully fine-tuned model.
  arXiv  Detail & Related papers  (2025-07-23T01:02:17Z)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning [93.00629872970364]
  Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks.<n>We introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions.<n>We study whether difficult problems -- those yielding no RL signals and mixed-quality reasoning traces -- can still be effectively used for training.
  arXiv  Detail & Related papers  (2025-06-05T07:53:59Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• POAR: Efficient Policy Optimization via Online Abstract State Representation Learning [6.171331561029968]
  State Representation Learning (SRL) is proposed to specifically learn to encode task-relevant features from complex sensory data into low-dimensional states. We introduce a new SRL prior called domain resemblance to leverage expert demonstration to improve SRL interpretations. We empirically verify POAR to efficiently handle tasks in high dimensions and facilitate training real-life robots directly from scratch.
  arXiv  Detail & Related papers  (2021-09-17T16:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.