Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation

• URL: http://arxiv.org/abs/2609.16937v1
• Date: Tue, 15 Sep 2026 10:12:56 GMT
• Title: Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation
• Abstract summary: On-policy distillation (OPD) has emerged as an effective approach for large language model post-training.<n>We propose $$OPD, which uses discounted temporal credit assignment to balance long-horizon supervision and optimization stability.<n>We develop a reward-compatible bounded mixing mechanism for $mathrmOPD$ that balances verifiable outcome feedback with the discounted OPD advantage.
• Score: 31.594936994611043
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: On-policy distillation (OPD) has emerged as an effective approach for large language model post-training, yet existing objectives face a trade-off between objective fidelity and optimization stability. Token-level OPD provides stable but local supervision, whereas sequence-level OPD captures future credit at the cost of horizon-dependent variance. We establish a unified temporal-credit view of these formulations, showing that practical token-level OPD can be interpreted as a temporal approximation to the sequence-level reverse-KL gradient. Building on this connection, we propose $γ$OPD, which uses discounted temporal credit assignment to balance long-horizon supervision and optimization stability, while admitting a horizon-independent variance bound. We further develop a reward-compatible bounded mixing (RBM) mechanism for $γ\mathrm{OPD}$ that balances verifiable outcome feedback with the discounted OPD advantage to move beyond purely teacher-dependent optimization. Experiments on mathematical and code reasoning demonstrate consistent improvements over existing OPD methods across vanilla, size-mismatched, and multi-teacher distillation settings.

Related papers

• Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization [16.249904972534114]
  We develop an $textbfA$DMM-$textbfI$nspired $textbfM$omentum (AIM) framework based on residual-penalty variable splitting.<n>AIM recovers the exponential moving average of a multiplier update from an ADMM-style multiplier update.<n>We propose $textbfA$ccelerated $textbfR$esidual (RADAR) to improve the update direction and momentum estimation.
  arXiv  Detail & Related papers  (2026-08-13T08:04:38Z)
• $β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation [85.07190808882523]
  On-policy self-distillation is a promising approach to improve reasoning language models.<n>We show that $$-OPSD consistently outperforms vanilla OPSD.
  arXiv  Detail & Related papers  (2026-07-30T17:41:16Z)
• Ratio-Variance Regularized Policy Optimization [64.95520246570446]
  We show that explicitly constraining the policy ratio variance provides a principled local approximation to trust-region constraints.<n>We introduce $bf R2bf VPO$ (Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal-dual optimization framework.
  arXiv  Detail & Related papers  (2026-05-26T09:53:42Z)
• $\boldsymbol{f}$-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control [10.758424473099055]
  Asynchronous execution is necessary for system efficiency, but structurally deviates from the ideal on-policy objective.<n>We propose f-OPD, a novel framework that adaptively regulates stale-sample influence and constrains policy drift.<n>Our results establish the first recipe for achieving a performance-efficiency trade-off in OPD, paving the way for long-horizon agentic post-training at scale.
  arXiv  Detail & Related papers  (2026-05-18T05:14:18Z)
• Holder Policy Optimisation [26.521180498291717]
  textbfHlderPO is a generalised policy optimisation framework.<n>It unifies token-level probability aggregation via the Hlder mean.<n>It achieves a state-of-the-art average accuracy of $54.9%$ across multiple mathematical benchmarks.
  arXiv  Detail & Related papers  (2026-05-12T12:45:03Z)
• Reducing Credit Assignment Variance via Counterfactual Reasoning Paths [11.514388061694268]
  Reinforcement learning for multi-step reasoning with large language models (LLMs) often relies on sparse terminal rewards.<n>We introduce a counterfactual comparison-based credit assignment framework, which samples multiple reasoning trajectories under the same input.<n>We construct an implicit process-level advantage estimator that transforms sparse terminal rewards into step-sensitive learning signals.
  arXiv  Detail & Related papers  (2026-04-20T13:33:54Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• Stabilizing Policy Optimization via Logits Convexity [59.242732612484474]
  We show that the convexity of the supervised fine-tuning loss with respect to model logits plays a key role in enabling stable training.<n>Motivated by this observation, we propose Logits Convex Optimization (LCO), a simple yet effective policy optimization framework.
  arXiv  Detail & Related papers  (2026-03-01T07:40:12Z)
• Unifying Stable Optimization and Reference Regularization in RLHF [64.16830602324345]
  This paper introduces a unified regularization approach that balances objectives of preventing reward hacking and maintaining stable policy updates.<n>Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity.
  arXiv  Detail & Related papers  (2026-02-12T03:31:19Z)
• Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models [7.316631310935769]
  Vision-Language-Action (VLA) models have shown strong generalization by leveraging large-scale demonstrations.<n>We propose Flow Policy Optimization (FPO) algorithm, which reformulates importance sampling by leveraging per-sample changes in the conditional flow-matching objective.<n>We evaluate FPO on the LIBERO benchmark and the ALOHA simulation task against supervised, preference-aligned, diffusion-based, autoregressive online RL.
  arXiv  Detail & Related papers  (2025-10-11T03:11:18Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.