Self-Distilled RLVR

• URL: http://arxiv.org/abs/2604.03128v2
• Date: Wed, 08 Apr 2026 08:22:36 GMT
• Title: Self-Distilled RLVR
• Abstract summary: We show that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training.<n>We propose textbfRLSD (textbfRLVR with textbfSelf-textbfDistillation)<n>This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.
• Score: 57.37526213765131
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from verifiable outcomes in the environment. Recently, the community has explored on-policy self-distillation (OPSD), where the same model serves as both teacher and student, with the teacher receiving additional privileged information such as reference answers to enable self-evolution. This paper demonstrates that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training. Accordingly, we identify the optimal niche for self-distillation and propose \textbf{RLSD} (\textbf{RL}VR with \textbf{S}elf-\textbf{D}istillation). Specifically, we leverage self-distillation to obtain token-level policy differences for determining fine-grained update magnitudes, while continuing to use RLVR to derive reliable update directions from environmental feedback (e.g., response correctness). This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.

Related papers

• RISE: Recursive Improvement via Self-Extrapolating Policy Distillation [47.92477203057629]
  We propose textbfRISE (textbfRecursive textbfImprovement via textbfSelf-textbfExtrapolating Policy Distillation)<n>RISE constructs a synthetic teacher directly from the model's own RLVR training trajectory.<n>Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation.
  arXiv  Detail & Related papers  (2026-09-04T15:47:51Z)
• On-policy Distillation with Verifiable Reward [32.72049308672042]
  On-policy distillation (OPD) andReinforcement Learning with Verifiable Rewards (RLVR) have become two widely adopted paradigms for post-training large language models.<n>We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR.<n> Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD.
  arXiv  Detail & Related papers  (2026-08-25T15:21:17Z)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance [50.77757355236912]
  We propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most.<n> Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and+3.05% on code.
  arXiv  Detail & Related papers  (2026-08-01T17:29:14Z)
• Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization [22.479229036646156]
  We propose a Privileged Tutoring Distillation Policy Optimization framework for Reinforcement Learning with Verifiable Rewards (RLVR)<n> PTD-PO provides dense guidance without exposing the answer to the student policy.<n> Experiments on LVLMs ranging from 2B to 8B parameters show that PTD-PO consistently outperforms RLVR and distillation baselines.
  arXiv  Detail & Related papers  (2026-06-05T07:43:22Z)
• Self-Distilled Agentic Reinforcement Learning [65.24201057390938]
  Self-Distilled Agentic Reinforcement Learning maps detached token-level signals into a sigmoid gate.<n>SDAR consistently outperforms hybrid RL--OPSD baselines across model scales.
  arXiv  Detail & Related papers  (2026-05-14T17:51:26Z)
• Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism [12.006669100411466]
  Current RLVR approaches rely heavily on accurate pseudo-label estimation.<n>We propose Dual Consensus Reinforcement Learning, which generates more reliable learning signals through a two-stage consensus mechanism.<n>We show that DCRL consistently improves Pass@1 over majority vote while yielding more stable training dynamics.
  arXiv  Detail & Related papers  (2026-03-17T07:57:23Z)
• Reinforcement-aware Knowledge Distillation for LLM Reasoning [63.53679456364683]
  Reinforcement learning (RL) post-training has recently driven gains in long chain-of-thought reasoning large language models (LLMs)<n>Most existing knowledge distillation methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization.<n>We propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update.
  arXiv  Detail & Related papers  (2026-02-26T00:20:39Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning [82.91265691530351]
  A$2$D is an Adaptive Ability Decomposing method for enhancing the effectiveness ofReinforcement Learning with verifiable rewards.<n>We first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions.<n>Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance.
  arXiv  Detail & Related papers  (2026-01-31T14:48:23Z)
• Reinforcement Learning via Self-Distillation [37.078107691613155]
  Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math.<n>Current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck.<n>We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO)<n>SDPO converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model.
  arXiv  Detail & Related papers  (2026-01-28T17:45:12Z)
• Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models [61.78513830395669]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs)<n>As models train longer and scale larger, more training prompts become residual prompts, those with zero variance rewards that provide no training signal.<n>We propose the Explore Residual Prompts in Policy Optimization framework, which encourages exploration on residual prompts and reactivates their training signals.
  arXiv  Detail & Related papers  (2025-11-06T20:40:27Z)
• Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values [53.72318444646282]
  We propose Reinforcement Learning with Explicit Human Values (RLEV)<n>RLEV aligns Large Language Model (LLM) optimization directly with quantifiable human value signals.<n>We show RLEV consistently outperforms correctness-only baselines across multiple RL algorithms and model scales.
  arXiv  Detail & Related papers  (2025-10-23T04:15:22Z)
• ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs [32.13266235550995]
  Reinforcement learning (RL) has become a standard paradigm for refining large language models (LLMs)<n>Inspired by observations from human learning, we introduce a RL technique that integrates verifiable outcomes with the model's own confidence estimates.
  arXiv  Detail & Related papers  (2025-09-22T13:00:35Z)
• Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA [26.079123341965687]
  We study low-rank learning and analyze how LoRA ranks and placements affect learning and forgetting.<n>A higher-rank LoRA improves task learning (plasticity) but increases forgetting, while a lower-rank LoRA enhances stability but limits adaptation.<n>Motivated by this, we propose Continual Dynamic Rank-Selective LoRA (CoDyRA), which continually updates PTMs with LoRA adapters of adaptively optimized ranks.
  arXiv  Detail & Related papers  (2024-12-01T23:41:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.