Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs

• URL: http://arxiv.org/abs/2603.22446v1
• Date: Mon, 23 Mar 2026 18:17:15 GMT
• Title: Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
• Abstract summary: Reinforcement learning with verifiable rewards (RLVR) has significantly improved reasoning in large language models (LLMs)<n>We present a systematic empirical study of RLVR's distributional effects organized around three main analyses.<n>We find that RL fine-tuning induces highly sparse and targeted changes, with only a small fraction of token distributions exhibiting meaningful divergence between the base and RL policies.
• Score: 65.76530158565903
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) has significantly improved reasoning in large language models (LLMs), yet the token-level mechanisms underlying these improvements remain unclear. We present a systematic empirical study of RLVR's distributional effects organized around three main analyses: (1) token-level characterization of distributional shifts between base and RL models, (2) the impact of token-level distributional shifts on sequence-level reasoning performance through cross-sampling interventions, and (3) fine-grained mechanics of these shifts at the token level. We find that RL fine-tuning induces highly sparse and targeted changes, with only a small fraction of token distributions exhibiting meaningful divergence between the base and RL policies. We further characterize the structure and evolution of these shifts through analyses of token entropy, positional concentration, and reallocation of probability mass. To assess the functional importance of these sparse changes, we conduct cross-sampling experiments that selectively swap token choices between the base and RL models with varying intervention budgets. We show that inserting only a small fraction of RL-sampled tokens into base generations progressively recovers RL performance gains, while injecting a similarly small number of base token choices into otherwise RL-generated sequences collapses performance to base levels, isolating a small set of token-level decisions directly responsible for RLVR's performance gains. Finally, we explore divergence-weighted variants of the advantage signal as a diagnostic intervention, finding that they can yield improvements over baselines. Together, our results shed light on the distributional changes induced by RLVR and provide a fine-grained, token-level lens for understanding RLVR fine-tuning as a targeted refinement process.

Related papers

• What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track [4.624097215050827]
  We introduce Fixed-SAE Track, a framework that trains one shared SAE per layer on activations pooled across the base model and all RL checkpoints.<n>We find that RL-induced drift is small, gradual, concept specific, and concentrated in late layers, mainly enhancing the sampling rates of a small set of ladder tokens.
  arXiv  Detail & Related papers  (2026-09-14T05:29:45Z)
• What are Key Factors for Updates in RL for LLM Reasoning? [41.37684066124904]
  We show that differences in off-policy degree, determined by the number of gradient steps per rollout, substantially affect the distribution of importance sampling ratios and their clipping behavior.<n>Motivated by these findings, we propose Adaptive Clip Policy Optimization (ACPO), which clipping boundaries across token groups according to the empirical variance of their importance sampling ratios.
  arXiv  Detail & Related papers  (2026-06-21T16:14:46Z)
• DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning [49.985643703496514]
  We propose DyCo-RL, which integrates dynamic cross-modal coordination into RLVR optimization.<n>Experiments demonstrate that DyCo-RL, when applied to Qwen2.5-VL-3B/7B, consistently improves four representative RLVR algorithms.
  arXiv  Detail & Related papers  (2026-06-06T07:56:25Z)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation [81.29776444949539]
  We argue that the direction of updates is a more critical lens for understanding RLVR's effects.<n>Our work establishes the direction of change as a key principle for analyzing and improving RLVR.
  arXiv  Detail & Related papers  (2026-03-23T15:42:24Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis [23.904569857346605]
  Reinforcement learning with computation rewards has become a standard post-training stage for boosting visual reasoning in vision-language models.<n>End-to-end benchmark gains conflate multiple factors, making it difficult to attribute improvements to specific skills.<n>We propose a Frankenstein-style analysis framework including: (i) functional localization via causal probing; (ii) update characterization via parameter comparison; and (iii) transferability test via model merging.
  arXiv  Detail & Related papers  (2026-02-12T20:44:27Z)
• Revisiting Entropy in Reinforcement Learning for Large Reasoning Models [54.96908589622163]
  We investigate the entropy dynamics of large language models trained withReinforcement learning with verifiable rewards (RLVR)<n>Our findings reveal that the number of off-policy updates, the diversity of training data, and the clipping thresholds in the optimization objective are critical factors influencing the entropy of LLMs trained with RLVR.
  arXiv  Detail & Related papers  (2025-11-08T12:50:41Z)
• Spotlight on Token Perception for Multimodal Reinforcement Learning [65.97597482517425]
  Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Vision-Language Models (LVLMs)<n>In this paper, we undertake a pioneering exploration of multimodal RLVR through the novel perspective of token perception.<n>We propose Visually-Perceptive Policy Optimization (VPPO), a novel policy gradient algorithm that explicitly leverages token perception to refine the learning signal.
  arXiv  Detail & Related papers  (2025-10-10T11:25:33Z)
• ASPO: Asymmetric Importance Sampling Policy Optimization [31.38346888572171]
  The Importance Sampling (IS) ratios of positive-advantage tokens are mismatched, leading to unbalanced token weighting for positive and negative tokens.<n>This mismatch suppresses the update of low-probability tokens while over-amplifying already high-probability ones.<n>We propose Asymmetric Importance Sampling Policy Optimization (ASPO), which uses a simple yet effective strategy that flips the IS ratios of positive-advantage tokens.
  arXiv  Detail & Related papers  (2025-10-07T15:54:24Z)
• Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning [106.68304931854038]
  Reinforcement learning with verifiable rewards (RLVR) has been widely used for enhancing the reasoning abilities of large language models (LLMs)<n>We conduct a systematic empirical analysis of the entropy-performance exchange mechanism of RLVR across different levels of granularity.<n>Our analysis reveals that, in the rising stage, entropy reduction in negative samples facilitates the learning of effective reasoning patterns.<n>In the plateau stage, learning efficiency strongly correlates with high-entropy tokens present in low-perplexity samples and those located at the end of sequences.
  arXiv  Detail & Related papers  (2025-08-04T10:08:10Z)
• One-Step Distributional Reinforcement Learning [10.64435582017292]
  We present the simpler one-step distributional reinforcement learning (OS-DistrRL) framework. We show that our approach comes with a unified theory for both policy evaluation and control. We propose two OS-DistrRL algorithms for which we provide an almost sure convergence analysis.
  arXiv  Detail & Related papers  (2023-04-27T06:57:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.