Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation

• URL: http://arxiv.org/abs/2607.10805v1
• Date: Sun, 12 Jul 2026 15:24:25 GMT
• Title: Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation
• Abstract summary: On-Policy Self-Distillation (OPSD) has emerged as a crucial paradigm for enhancing and aligning Large Language Models (LLMs)<n>OPSD paradoxically degrades downstream performance in complex reasoning tasks.<n>We propose textbf Dual-Perspective OPSD (AD-OPSD), a robust control framework that dynamically moderates the self-distillation objective.
• Score: 21.932079347631632
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-Policy Self-Distillation (OPSD) has emerged as a crucial paradigm for enhancing and aligning Large Language Models (LLMs). However, in complex reasoning tasks, OPSD paradoxically degrades downstream performance. In this paper, we systematically investigate this pathology and identify a severe optimization trap we define as \textbf{Thinking Collapse} -- a sharp decline in the model's native intermediate reasoning behavior, measured by epistemic-token density (ET per 1k). Through entropy-based gradient masking and token-level target analysis, we show that this collapse is triggered by aggressive teacher gradients at high-student-entropy decision forks, where student epistemic tokens are frequently suppressed into teacher non-epistemic targets and are highly concentrated in high pointwise student-teacher divergence regions. To resolve this optimization pathology, we propose \textbf{Adaptive Dual-Perspective OPSD (AD-OPSD)}, a robust control framework that dynamically moderates the self-distillation objective. AD-OPSD selectively anchors high-suppression-risk sandboxed tokens to a reference prior derived from the frozen base model via an asymmetrical pointwise divergence gate, preserving native thinking capacity while retaining OPSD's error-correcting power. Extensive experiments across competitive mathematical benchmarks show that AD-OPSD improves over standard OPSD by up to \textbf{+4.1\%} absolute average accuracy across diverse model scales and datasets. Further analysis demonstrates that AD-OPSD mitigates thinking collapse and generalizes robustly to different post-training paradigms.

Related papers

• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
  We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
  arXiv  Detail & Related papers  (2026-07-02T14:33:07Z)
• RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation [46.554172283247915]
  On-policy self-distillation provides dense, token-level supervision for reasoning models.<n>We show that the learning signal drawn from this distributional gap concentrates on style tokens rather than task-bearing ones.<n>We propose textbfRLCSD (Reinforcement Learning with Contrastive on-policy Self-Distillation) to mitigate this drift.
  arXiv  Detail & Related papers  (2026-06-10T06:31:59Z)
• Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning [41.384652481442735]
  We propose textbfDirection-Adaptive Self-Distillation (textbfDASD), which reframes privileged self-distillation from uniform teacher imitation into entropy-routed directional supervision.<n>Across six mathematical reasoning benchmarks, DASD achieves the best macro Avg@16 over strong RLVR and self-distillation baselines.
  arXiv  Detail & Related papers  (2026-05-21T10:07:46Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Models [54.16797570104461]
  Diffusion-based language models (dLLMs) have emerged as a promising alternative to autoregressive language models.<n>We investigate non-autoregressive decoding in dLLMs by systematically analyzing its inference dynamics along the temporal axis.
  arXiv  Detail & Related papers  (2026-04-12T10:26:41Z)
• Hindsight-Anchored Policy Optimization: Turning Failure into Feedback in Sparse Reward Settings [7.554296968064274]
  We present Hindsight-Anchored Policy optimization (HAPO) for Reinforcement Learning with Verifiable Rewards (RLVR)<n>HAPO employs a hindsight mechanism that selectively anchors optimization to teacher demonstrations during failure.<n>We show that HAPO achieves textitasymptotic consistency: by naturally annealing the teacher signal as the policy improves, HAPO recovers the unbiased on-policy gradient.
  arXiv  Detail & Related papers  (2026-03-11T21:33:41Z)
• Interpretable Maximum Margin Deep Anomaly Detection [10.25312830872628]
  Anomaly detection is a crucial machine-learning task with wide-ranging applications.<n>We propose Maximum Interpretable Margin Deep Anomaly Detection (IMD-AD)
  arXiv  Detail & Related papers  (2026-03-07T07:06:45Z)
• Amortized Reasoning Tree Search: Decoupling Proposal and Decision in Large Language Models [2.5170433424424874]
  Reinforcement Learning with Verifiable Rewards has established itself as the dominant paradigm for instilling rigorous reasoning capabilities in Large Language Models.<n>We identify a critical pathology in this alignment process: the systematic suppression of valid but rare (low-likelihood under the base model distribution) reasoning paths.<n>We propose Amortized Reasoning Tree Search (ARTS) to counteract this collapse without discarding the base model's latent diversity.
  arXiv  Detail & Related papers  (2026-02-13T11:52:50Z)
• On the Plasticity and Stability for Post-Training Large Language Models [54.757672540381236]
  We identify a root cause as the conflict between plasticity and stability gradients.<n>We propose Probabilistic Conflict Resolution (PCR), a framework that models gradients as random variables.<n>PCR significantly smooths the training trajectory and achieves superior performance in various reasoning tasks.
  arXiv  Detail & Related papers  (2026-02-06T07:31:26Z)
• On The Presence of Double-Descent in Deep Reinforcement Learning [43.22339935902436]
  Double descent (DD) paradox remains largely unexplored in the non-stationary domain of Deep Reinforcement Learning (DRL)<n>We present preliminary evidence that DD exists in model-free DRL, investigating it systematically across varying model capacity using the Actor-Critic framework.<n>These findings establish DD as a factor in DRL and provide an information-based mechanism for designing agents that are more general, transferable, and robust.
  arXiv  Detail & Related papers  (2025-11-10T09:45:03Z)
• Improving Domain Generalization in Self-supervised Monocular Depth Estimation via Stabilized Adversarial Training [61.35809887986553]
  We propose a general adversarial training framework, named Stabilized Conflict-optimization Adversarial Training (SCAT) SCAT integrates adversarial data augmentation into self-supervised MDE methods to achieve a balance between stability and generalization. Experiments on five benchmarks demonstrate that SCAT can achieve state-of-the-art performance and significantly improve the generalization capability of existing self-supervised MDE methods.
  arXiv  Detail & Related papers  (2024-11-04T15:06:57Z)
• Towards Understanding the Robustness of Diffusion-Based Purification: A Stochastic Perspective [65.10019978876863]
  Diffusion-Based Purification (DBP) has emerged as an effective defense mechanism against adversarial attacks.<n>In this paper, we propose that the intrinsicity in the DBP process is the primary factor driving robustness.
  arXiv  Detail & Related papers  (2024-04-22T16:10:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.