Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models

• URL: http://arxiv.org/abs/2604.08527v1
• Date: Thu, 09 Apr 2026 17:58:02 GMT
• Title: Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
• Abstract summary: On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers.<n>As training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data.<n>This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance.<n>We propose StableOPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation.
• Score: 35.70546523782139
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers. We identify a failure mode of OPD: as training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data. This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance. We attribute this problem to the interaction between student-induced data collection and the distillation objective, which implicitly favors long and repetitive rollouts. To address this issue, we propose StableOPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation. These together mitigate repetition-induced length inflation and further stabilize OPD training. Across multiple math reasoning datasets, our approach prevents truncation collapse, stabilizes training dynamics, and improves performance by 7.2% on average.

Related papers

• TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs [83.78708832901194]
  Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild.<n>However, real-world deployment remains challenging: modalities may be missing or noisy at test time.<n>We propose a solution in which a frozen teacher guides an adaptive low-rank student via self-distillation.<n>Experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation.
  arXiv  Detail & Related papers  (2026-08-18T23:40:28Z)
• CurateEvo: Data-Curation Evolving for Agentic Post-Training [53.68149869349268]
  We propose CurateEvo, a failure-driven dynamic evolution framework for agentic post-training data curation.<n>CurateEvo represents the curation strategy as executable code and iteratively rewrites it using failed trajectories from a held-out development set.<n>At each epoch, the evolved strategy transforms a fixed raw corpus into supervised fine-tuning data, reinforcement learning data, and an inference-time memory bank.
  arXiv  Detail & Related papers  (2026-07-07T11:07:00Z)
• $\oldsymbol{f}$-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control [10.758424473099055]
  Asynchronous execution is necessary for system efficiency, but structurally deviates from the ideal on-policy objective.<n>We propose f-OPD, a novel framework that adaptively regulates stale-sample influence and constrains policy drift.<n>Our results establish the first recipe for achieving a performance-efficiency trade-off in OPD, paving the way for long-horizon agentic post-training at scale.
  arXiv  Detail & Related papers  (2026-05-18T05:14:18Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Elucidating Representation Degradation Problem in Diffusion Model Training [16.02291234220539]
  Elucidated Representation Diffusion (ERD) is a plug-and-play framework that dynamically reallocates optimization effort according to effective recoverability.<n>By stabilizing representation learning without external supervision, ERD accelerates convergence and achieves strong empirical performance across diffusion backbones.
  arXiv  Detail & Related papers  (2026-05-11T16:21:45Z)
• VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation [14.33516456804297]
  On-Policy VLA Distillation (VLA-OPD) is a framework bridging the efficiency of online Reinforcement Learning (RL) with the robustness of offline Supervised Fine-Tuning (SFT)
  arXiv  Detail & Related papers  (2026-03-27T17:59:33Z)
• Analyzing and Improving Diffusion Models for Time-Series Data Imputation: A Proximal Recursion Perspective [45.713195454899875]
  Diffusion models (DMs) have shown promise for Time-Series Data Imputation.<n>DMs' performance remains inconsistent in complex scenarios.<n>We propose a novel framework called SPIRIT (Semi-Proximal Transport Regularized time-series Imputation)
  arXiv  Detail & Related papers  (2026-02-01T12:11:57Z)
• Entropy Production in Machine Learning Under Fokker-Planck Probability Flow [0.0]
  We propose an entropy-based retraining framework grounded in non-equilibrium cost dynamics.<n>We show that entropy-triggered retraining achieves predictive performance comparable to high-frequency retraining.
  arXiv  Detail & Related papers  (2026-01-02T04:01:57Z)
• Stabilizing Reinforcement Learning with LLMs: Formulation and Practices [61.361819972410046]
  We show why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE.<n>This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training.
  arXiv  Detail & Related papers  (2025-12-01T07:45:39Z)
• Balanced Actor Initialization: Stable RLHF Training of Distillation-Based Reasoning Models [27.0496567592082]
  The development of alignment and reasoning capabilities in large language models has seen remarkable progress.<n>The third paradigm of applying RLHF to distillation-trained models presents significant challenges.<n>We propose Balanced Actor Initialization (BAI), a two-stage weighted model merging approach.
  arXiv  Detail & Related papers  (2025-08-30T01:53:25Z)
• Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning [0.0]
  Fine-tuning foundation models via reinforcement learning (RL) has proven promising for aligning to downstream objectives. We exploit the hierarchical nature of diffusion models (DMs) and train them dynamically at each epoch with a tailored RL method. We show that models trained with HRF achieve better preservation of diversity in downstream tasks, thus enhancing the fine-tuning robustness and at uncompromising mean rewards.
  arXiv  Detail & Related papers  (2024-10-10T19:06:23Z)
• Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning [47.64252639582435]
  We focus on semi-supervised continual learning (SSCL), where the model progressively learns from partially labeled data with unknown categories.<n>We propose a novel approach called Dynamic Sub-Graph Distillation (DSGD) for semi-supervised continual learning.
  arXiv  Detail & Related papers  (2023-12-27T04:40:12Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.