Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation

• URL: http://arxiv.org/abs/2608.09263v1
• Date: Mon, 10 Aug 2026 08:18:48 GMT
• Title: Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
• Abstract summary: Outcome verifiers score completed reasoning traces but do not assign credit to intermediate tokens.<n> Privileged self-distillation attempts to fill this gap by rescoring a model's own rollout with training-only information.<n>We separate three questions: whether the score tracks better actions, whether feedback construction changes what is compared, and what behavior the training loss reinforces.
• Score: 41.01488608247856
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Outcome verifiers score completed reasoning traces but do not assign credit to intermediate tokens. Privileged self-distillation attempts to fill this gap by rescoring a model's own rollout with training-only information. A token likelihood change, however, is not automatically outcome credit. We separate three questions: whether the score tracks better actions, whether feedback construction changes what is compared, and what behavior the training loss reinforces. We establish these distinctions formally. When a rollout is scored using hindsight feedback written about that same rollout, its content determines both the tokens and the scoring context, creating direct self-dependence. Using feedback from another rollout of the same problem removes this dependence but does not guarantee a useful score. In matched experiments with a 20B model on AIME 2025, the implemented additive score is near chance (AUC=0.505) and slightly favors incorrect traces after length adjustment. In the paired comparison, the outcome-only control records 64.2\%, versus 24.2\%--33.9\% for five token-score variants. The results motivate validating score meaning, feedback construction, and training behavior separately before calling a likelihood signal credit.

Related papers

• Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format [0.0]
  We compare probabilities obtained by scoring answer tokens with predictions generated after a written rationale.<n>Across 13 model-domain cells covering four retail tasks, the scored readout ranks outcomes more accurately in 12 of 13 cells.
  arXiv  Detail & Related papers  (2026-09-09T08:37:46Z)
• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay [0.6768558752130311]
  We audit against policy-conditional ground truth from executed replay in a single-agent tool environment.<n>No step-level credit signals we audit show reliable incremental fidelity beyond marginal-matched shuffled control.<n>A confidence-only router recovers pivotal steps at chance level, but cuts judge cost by 13.1% per turn.
  arXiv  Detail & Related papers  (2026-08-20T08:04:00Z)
• When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit [2.823425836233287]
  Re-simulation benchmarks may use reference-conditioned forgiveness, under which an agent receives credit when a logged human reference fails a compliance channel.<n>We audit this risk in NAVSIM v2.2 original scene single-stage scoring.
  arXiv  Detail & Related papers  (2026-08-05T14:22:23Z)
• Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters [10.230437172157474]
  Chain-of-thought prompting improves LLM reasoning, but the source is contested.<n>We repeatedly sample each model on the same question and pair a shorter with a longer of its own natural generations that follow the same reasoning plan.<n>We find that the extra tokens leave accuracy essentially unchanged for every independently-trained reasoner.<n>We ask whether two traces expressing the same semantic content produce different outcomes when one is more verbose.
  arXiv  Detail & Related papers  (2026-06-29T11:06:46Z)
• CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning [4.663685189987781]
  Self-distilled agentic reinforcement learning augments trajectory-level reward with a token-level distillation loss.<n>The prevailing recipe gates this loss by a single scalar, the teacher-student log-probability gap.<n>We introduce CRAFT, a threepillar credit-assignment scheme that addresses both limitations.
  arXiv  Detail & Related papers  (2026-06-28T16:11:47Z)
• Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation [0.0]
  Per-token counterfactual credit estimation asks which token in a language-model rollout caused the final answer to be right or wrong.<n>We measure what that assumption costs on a stock inference engine, with a three-pass design.<n>Across six configurations and three models, re-feeding changes the credit estimate at rates 14-28 percentage points above the replica floor.
  arXiv  Detail & Related papers  (2026-06-14T06:09:16Z)
• CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization [50.59956036193097]
  A model produces a correct solution under reinforcement learning with verifiable rewards (RLVR)<n>Every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler.<n>We propose Contrastive Evidence Policy Optimization (CEPO)<n>CEPO asks a sharper question at every token: "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?"
  arXiv  Detail & Related papers  (2026-05-19T06:46:19Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits [56.379030343338776]
  We study criticfree RL from a token-level perspective, revealing the token-flipping phenomenon.<n>We show that a token's change in probability is not fully determined by its own advantage; coupled gradient interactions with other tokens also play a non-negligible role.
  arXiv  Detail & Related papers  (2026-05-09T04:07:20Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Integrating Rankings into Quantized Scores in Peer Review [61.27794774537103]
  In peer review, reviewers are usually asked to provide scores for the papers. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are no standard procedure for using this ranking information and Area Chairs may use it in different ways. We take a principled approach to integrate the ranking information into the scores.
  arXiv  Detail & Related papers  (2022-04-05T19:39:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.