Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation

• URL: http://arxiv.org/abs/2605.09253v1
• Date: Sun, 10 May 2026 01:41:43 GMT
• Title: Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
• Abstract summary: We investigate high-loss tokens, a token type that--as the most direct signal of student-teacher mismatch under On-Policy Distillation's per-token KL objective--should progressively diminish as training converges according to existing studies.<n>We find that these tokens provide negligible functional contribution to the model's actual reasoning performance.
• Score: 4.624042537090342
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While recent work in Reinforcement Learning with Verifiable Rewards (RLVR) has shown that a small subset of critical tokens disproportionately drives reasoning gains, an analogous token-level understanding of On-Policy Distillation (OPD) remains largely unexplored. In this work, we investigate high-loss tokens, a token type that--as the most direct signal of student-teacher mismatch under OPD's per-token KL objective--should progressively diminish as training converges according to existing studies; however, our empirical analysis shows otherwise. Even after OPD training reaches apparent saturation, a substantial subset of tokens continues to exhibit persistently high loss; these tokens, which we term Rock Tokens, can account for up to 18\% of the tokens in generated outputs. Our investigation reveals two startling paradoxes. First, despite their high occurrence frequency providing a disproportionately large share of total gradient norms, Rock Tokens themselves remain stagnant throughout training, resisting teacher-driven corrections. Second, through causal intervention, we find that these tokens provide negligible functional contribution to the model's actual reasoning performance. These findings suggest that a vast amount of optimization bandwidth is spent on structural and discourse residuals that the student model cannot or need not internalize. By deconstructing these dynamics, we demonstrate that strategically bypassing these ``stumbling blocks'' can significantly streamline the alignment process, challenging the necessity of uniform token weighting and offering a more efficient paradigm for large-scale model distillation.

Related papers

• Extremely Sparse Supervision Incentivizes Reasoning Ability [13.22207761303279]
  Large language models demonstrate increasingly strong reasoning capabilities through effective post-training.<n> prevailing post-training methods optimize over massive numbers of tokens, implicitly assuming that effective learning must be token-intensive.<n>We discover a counter-intuitive phenomenon: reasoning can be effectively incentivized by an extremely small fraction of generated tokens.
  arXiv  Detail & Related papers  (2026-09-03T23:38:48Z)
• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation [59.43605629311855]
  We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits.<n>Our primary contribution is therefore a gradient-level characterization of reverse-KL OPD trained with the K2 estimator, with SuRe as one empirical instantiation.
  arXiv  Detail & Related papers  (2026-08-26T11:14:42Z)
• PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping [5.89473045822308]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective paradigm for improving the reasoning capability of Large Vision-Language Models (LVLMs)<n>Existing RLVR methods rely on trajectory-level outcome rewards, which assign identical learning signals across all generated tokens.<n>We propose Perception-Reinforced Policy Optimization (PRPO), a token-level reinforcement learning framework that explicitly identifies and reinforces pivotal perceptual tokens.
  arXiv  Detail & Related papers  (2026-06-07T16:06:29Z)
• Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy [64.72195169444738]
  Policy-gradient methods treat each token in a trajectory equally, leading to uniform credit assignment.<n>We show that such uniform credit assignment largely misallocates token-level training signals.<n>We propose an embarrassingly simple token reweighting approach, ActFocus, that downweights gradients on reasoning tokens.
  arXiv  Detail & Related papers  (2026-05-14T08:33:02Z)
• The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits [56.379030343338776]
  We study criticfree RL from a token-level perspective, revealing the token-flipping phenomenon.<n>We show that a token's change in probability is not fully determined by its own advantage; coupled gradient interactions with other tokens also play a non-negligible role.
  arXiv  Detail & Related papers  (2026-05-09T04:07:20Z)
• Context Bootstrapped Reinforcement Learning [51.213972559315486]
  Reinforcement Learning from Verifiable Rewards (RLVR) suffers from exploration inefficiency.<n>We propose Contextped Bootstrapped Reinforcement Learning (CBRL), which augments RLVR training by prepending few-shot demonstrations to training prompts.<n>CBRL consistently improves success rate, provides better exploration efficiency, and is algorithm-agnostic.
  arXiv  Detail & Related papers  (2026-03-19T14:23:59Z)
• Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning [64.04741347596938]
  We introduce Token Hidden Reward (THR), a token-level metric that quantifies each token's influence on the likelihood of correct responses.<n>We find that training dynamics are dominated by a small subset of tokens with high absolute THR values.<n>This insight suggests a THR-guided reweighting algorithm that modulates GRPO's learning signals to explicitly bias training toward exploitation or exploration.
  arXiv  Detail & Related papers  (2025-10-04T04:49:44Z)
• Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruning [62.23671919314693]
  Large language models (LLMs) have demonstrated significant improvements in contextual understanding.<n>However, their ability to attend to truly critical information during long-context reasoning and generation still falls behind the pace.<n>We introduce a two-stage framework called Learning to Focus (LeaF) to mitigate confounding factors.
  arXiv  Detail & Related papers  (2025-06-09T15:16:39Z)
• ToDi: Token-wise Distillation via Fine-Grained Divergence Control [9.958797874295355]
  Token-wise Distillation (ToDi) is a novel method that adaptively combines Forward KL and Reverse KL per token using a sigmoid-based weighting function.<n>ToDi consistently outperforms recent distillation baselines using uniform or less granular strategies.
  arXiv  Detail & Related papers  (2025-05-22T06:51:16Z)
• Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability [53.51560766150442]
  Critical tokens are elements within reasoning trajectories that significantly influence incorrect outcomes.<n>We present a novel framework for identifying these tokens through rollout sampling.<n>We show that identifying and replacing critical tokens significantly improves model accuracy.
  arXiv  Detail & Related papers  (2024-11-29T18:58:22Z)
• Keypoint-based Progressive Chain-of-Thought Distillation for LLMs [46.53906673648466]
  Chain-of-thought distillation is a powerful technique for transferring reasoning abilities from large language models to smaller student models. Previous methods typically require the student to mimic the step-by-step rationale produced by LLMs. We propose a unified framework, called KPOD, to address these issues.
  arXiv  Detail & Related papers  (2024-05-25T05:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.