Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers

• URL: http://arxiv.org/abs/2606.12507v1
• Date: Wed, 10 Jun 2026 17:53:19 GMT
• Title: Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers
• Abstract summary: We propose a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student.<n>Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves satisfaction comparable to judge-based rubricO.
• Score: 25.040339775169883
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rubrics have emerged as an alternative to RLVR in open-ended domains where a single ground-truth final answer is not available. Existing rubric-based training methods rely on an LLM verifier that scores each rollout against rubrics. This introduces substantial training-time overhead, exposes optimization to verifier-specific biases, and reduces rubric feedback to a sparse end-of-trajectory signal. We propose Rubric-Guided Self-Distillation (RGSD), a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student. RGSD distills the rubric-conditioned teacher distribution into the student token-by-token, replacing sparse trajectory-level rewards with dense per-token learning signals and removing the LLM judge from the training loop entirely. Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves rubric satisfaction comparable to judge-based GRPO while using one on-policy rollout per prompt and no training-time verifier calls. Ablations show that raw rubrics provide a stronger teacher enrichment signal than self-generated reference responses, while a stronger GRPO judge can outperform RGSD in some settings, positioning RGSD as a complementary verifier-free alternative when verifier cost or reliability is the bottleneck.

Related papers

• Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning [19.29382518431932]
  Group-Calibrated On-Policy Distillation (GC-OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher.<n>Group-Calibrated On-Policy Distillation (GC-OPD) normalizes verifier rewards and trajectory-level OPD scores within each rollout group.<n>Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages.
  arXiv  Detail & Related papers  (2026-08-19T17:54:58Z)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance [50.77757355236912]
  We propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most.<n> Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and+3.05% on code.
  arXiv  Detail & Related papers  (2026-08-01T17:29:14Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards [68.15832368034815]
  We present a framework that co-designs queries and rubrics.<n>QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline.
  arXiv  Detail & Related papers  (2026-06-02T17:53:04Z)
• ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents [48.80766702702854]
  LLM-based search agents are trained predominantly with outcome-only reward, leaving the search process itself unsupervised.<n>This signal degenerates on outcome-homogeneous groups where all sampled trajectories share the same correctness, yielding zero within-group advantage and no gradient.<n>We propose ARBOR (Adaptive Buffer for Online Reward), a reusable process-reward framework that maintains a rubric memory shared across queries.
  arXiv  Detail & Related papers  (2026-06-02T06:58:54Z)
• CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO [9.443660785229719]
  Reinforcement learning with verifiable rewards (RLVR) has been widely used to improve reasoning in large language models.<n>This work proposes CAST, an answer-free self-distillation method for GRPO-style RLVR.
  arXiv  Detail & Related papers  (2026-05-29T13:21:30Z)
• OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Models [12.17078443900398]
  On-Policy Self-Distillation (OPSD) has recently emerged as an alternative to Reinforcement Learning with Verifiable Rewards (RLVR)<n>We show that in thinking-enabled mathematical reasoning, OPSD behaves most reliably as a compression mechanism rather than a correction mechanism.
  arXiv  Detail & Related papers  (2026-05-07T13:04:34Z)
• Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision [50.61441331643804]
  Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training.<n>Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations.<n>We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations.
  arXiv  Detail & Related papers  (2026-04-13T19:46:55Z)
• Self-Distilled RLVR [57.37526213765131]
  We show that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training.<n>We propose textbfRLSD (textbfRLVR with textbfSelf-textbfDistillation)<n>This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.
  arXiv  Detail & Related papers  (2026-04-03T15:50:07Z)
• What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time [57.533031432715084]
  Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams.<n>Existing TTRL methods rely exclusively on positive pseudo-labeling strategies.<n>We propose SCRL, a robust test-time reinforcement learning framework that effectively mitigates label noise amplification.
  arXiv  Detail & Related papers  (2026-03-20T11:47:12Z)
• Refinement Provenance Inference: Detecting LLM-Refined Training Prompts from Model Behavior [58.751981587234916]
  This paper formalizes the Refinement Provenance Inference (RPI) audit task as Refinement Provenance Inference (RPI)<n>We propose RePro, a logit-based framework that fuses teacher-forced likelihood features with logit-ranking signals.<n>During training, RePro learns a transferable representation via shadow fine-tuning, and uses a lightweight linear head to infer provenance on unseen victims without training-data access.
  arXiv  Detail & Related papers  (2026-01-05T10:16:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.