Related papers: Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation

Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation

URL: http://arxiv.org/abs/2606.19327v1
Date: Wed, 17 Jun 2026 17:54:04 GMT
Title: Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
Abstract summary: We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
Score: 60.55792673956761
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved. We propose \textbf{Rubric-Conditioned Self-Distillation}, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student's own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization. We instantiate this framework with a two-stage pipeline that first learns to generate task-specific rubrics and then trains a rubric-guided reasoner. We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process, surpassing GRPO by 1.0 points and OPSD by 0.9 points on average.

Related papers

Adaptive Supervised Anchoring for On-Policy Self-Distillation [15.445625053566244]
On-policy self-distillation adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student.<n>We show that when student rollouts drift from target trajectories, conditioning the teacher on off-target prefixes substantially weakens its task-relevant supervision.<n>We propose a unified training framework that separates two complementary supervision pathways.
arXiv Detail & Related papers (2026-08-08T05:46:32Z)
H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
arXiv Detail & Related papers (2026-07-21T10:47:27Z)
Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
arXiv Detail & Related papers (2026-07-02T14:33:07Z)
The Role of Feedback Alignment in Self-Distillation [0.3580891736370874]
Self-distillation trains the model to retain this improvement when the context is not present.<n>We study context design for self-distillation by training a solver on feedback from a frozen critic.
arXiv Detail & Related papers (2026-06-09T17:50:09Z)
Distilling LLM Feedback for Lean Theorem Proving [20.86168010460303]
Post-training for reasoning models typically combines supervised fine-tuning with reinforcement learning from verifiable rewards.<n>This algorithm suffers from sparse rewards, limited exploration, and mode collapse.<n>We propose Feedback Distillation, a training method where the model is trained to match, at the token level, its own distribution conditioned on privileged feedback.
arXiv Detail & Related papers (2026-05-29T05:35:00Z)
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
arXiv Detail & Related papers (2026-05-13T15:05:30Z)
From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation [22.436966302243565]
On-policy self-distillation has emerged as a promising paradigm for post-training language models.<n>We show that the self-distillation token reward is a Bayesian filtering increment whose trajectory sum is exactly the pointwise mutual information between the response and the feedback given the input.<n>We propose CREDIT, which isolates the input-specific component with a batch-contrastive baseline.
arXiv Detail & Related papers (2026-05-12T06:43:17Z)
RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time [51.256131853751754]
We show that teaching reward models to produce explicit, multi-dimensional critiques transforms them from passive evaluators into active optimization tools.<n>We introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data.<n>The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models.
arXiv Detail & Related papers (2026-04-13T15:38:09Z)
In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback [38.915062716409686]
InTRO is a new framework that enables both token-level exploration and self-feedback for accurate and concise reasoning.<n>InTRO consistently outperforms other baselines, raising solution accuracy by up to 20% relative to the base model.<n>Its chains of thought are notably more concise, exhibiting reduced verbosity.
arXiv Detail & Related papers (2025-11-13T01:47:06Z)
Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning [50.20267980386502]
We learn a dense, token-level reward model for process supervision directly from expert demonstrations.<n>The learned reasoning reward serves two complementary roles: (i) it provides step-level feedback to optimise a reasoning policy during training; and (ii) it functions at inference as a critic to rerank sampled traces under fixed compute budgets.
arXiv Detail & Related papers (2025-10-02T09:55:26Z)
RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning [64.46921169261852]
RAG-Zeval is a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task.<n>Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments.<n>Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments.
arXiv Detail & Related papers (2025-05-28T14:55:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.