HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation

• URL: http://arxiv.org/abs/2603.10359v1
• Date: Wed, 11 Mar 2026 03:12:10 GMT
• Title: HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation
• Abstract summary: We propose Hindsight Entropy-Assisted Learning (HEAL), an RL-free framework designed to bridge this reasoning gap.<n>HEAL synergizes three core modules: (1) Guided Entropy-Assisted Repair (GEAR), an active intervention mechanism that detects critical reasoning breakpoints via entropy dynamics and injects targeted hindsight hints to repair broken trajectories; (2) Perplexity-Uncertainty Ratio Estimator (PURE), a rigorous filtering protocol that decouples genuine cognitive breakthroughs from spurious shortcuts; and (3) Progressive Answer-guided Curriculum Evolution (PACE), a three-stage distillation strategy that organizes training from foundational alignment to
• Score: 19.13058927652914
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distilling reasoning capabilities from Large Reasoning Models (LRMs) into smaller models is typically constrained by the limitation of rejection sampling. Standard methods treat the teacher as a static filter, discarding complex "corner-case" problems where the teacher fails to explore valid solutions independently, thereby creating an artificial "Teacher Ceiling" for the student. In this work, we propose Hindsight Entropy-Assisted Learning (HEAL), an RL-free framework designed to bridge this reasoning gap. Drawing on the educational theory of the Zone of Proximal Development(ZPD), HEAL synergizes three core modules: (1) Guided Entropy-Assisted Repair (GEAR), an active intervention mechanism that detects critical reasoning breakpoints via entropy dynamics and injects targeted hindsight hints to repair broken trajectories; (2) Perplexity-Uncertainty Ratio Estimator (PURE), a rigorous filtering protocol that decouples genuine cognitive breakthroughs from spurious shortcuts; and (3) Progressive Answer-guided Curriculum Evolution (PACE), a three-stage distillation strategy that organizes training from foundational alignment to frontier breakthrough. Extensive experiments on multiple benchmarks demonstrate that HEAL significantly outperforms traditional SFT distillation and other baselines.

Related papers

• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
  We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
  arXiv  Detail & Related papers  (2026-07-02T14:33:07Z)
• IDLM: Inverse-distilled Diffusion Language Models [70.5793829229702]
  We extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting.<n>From a theoretical perspective, the inverse distillation objective lacks uniqueness guarantees, which may lead to suboptimal solutions.<n>We show that our method, Inverse-distilled Diffusion Language Models (IDLM), reduces the number of inference steps by 4x-64x.
  arXiv  Detail & Related papers  (2026-02-22T06:47:04Z)
• Native Reasoning Models: Training Language Models to Reason on Unverifiable Data [16.065264121785294]
  We introduce NRT (Native Reasoning Training), a novel framework that cultivates complex reasoning.<n>NRT reframes the training problem by treating the reasoning process as a latent variable.<n>NRT achieves state-of-the-art performance among verifier-free methods.
  arXiv  Detail & Related papers  (2026-02-12T04:15:46Z)
• KEPO: Knowledge-Enhanced Preference Optimization for Reinforcement Learning with Reasoning [24.072603982041798]
  Reinforcement learning has emerged as a promising paradigm for inducing explicit reasoning behaviors in large language and vision-language models.<n>However, reasoning-oriented RL post-training remains fundamentally challenging due to sparse trajectory-level rewards.<n>Recent on-policy distillation methods introduce dense teacher supervision to stabilize optimization, but apply it uniformly across all generated trajectories.
  arXiv  Detail & Related papers  (2026-01-30T23:28:37Z)
• Latent-Space Contrastive Reinforcement Learning for Stable and Efficient LLM Reasoning [16.244366307890832]
  We propose textbfDeepLatent Reasoning (DLR), a latent-space bidirectional contrastive reinforcement learning framework.<n>This framework shifts the trial-and-error cost from expensive token-level full sequence generation to the continuous latent manifold.<n> Experiments demonstrate that DLR achieves more stable training convergence, supports longer-horizon reasoning chains, and facilitates the sustainable accumulation of reasoning capabilities.
  arXiv  Detail & Related papers  (2026-01-24T03:18:22Z)
• Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning [48.041170200238206]
  We introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source reasoning model.<n>It achieves SOTA performance among open-source models of comparable scale across challenging benchmarks in mathematics, scientific reasoning, and code generation.
  arXiv  Detail & Related papers  (2026-01-14T02:43:17Z)
• The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solving [57.652356955571065]
  State-of-the-art large language model (LLM) pipelines rely on bootstrapped reasoning loops.<n>We analyze how this design choice is sensitive to the collapse of the model's distribution over reasoning paths.<n>We introduce Distributional Creative Reasoning (DCR), a unified variational objective that casts training as gradient flow through probability measures on solution traces.
  arXiv  Detail & Related papers  (2026-01-02T17:10:31Z)
• From Reasoning LLMs to BERT: A Two-Stage Distillation Framework for Search Relevance [20.096802351171377]
  e-commerce search systems face strict latency requirements that prevent the direct application of Large Language Models.<n>We propose a two-stage reasoning distillation framework to transfer reasoning capabilities from a powerful teacher LLM to a lightweight, deployment-friendly student model.<n>Our framework achieves significant improvements across multiple metrics, validating its effectiveness and practical value.
  arXiv  Detail & Related papers  (2025-10-13T06:46:43Z)
• Reverse-Engineered Reasoning for Open-Ended Generation [50.35487467267016]
  We introduce REverse-Engineered Reasoning (REER)<n>Instead of building a reasoning process forwards'' through trial-and-error or imitation, REER works backwards'' from known-good solutions to discover latent, step-by-step deep reasoning process that could have produced them.<n>Our model, DeepWriter-8B, trained on this data, not only surpasses strong open-source baselines but also achieves performance competitive with, and at times superior to, leading proprietary models like GPT-4o and Claude 3.5.
  arXiv  Detail & Related papers  (2025-09-07T18:07:58Z)
• ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Models [76.28894983518164]
  Small Language Models (SLMs) are a cost-effective alternative to Large Language Models (LLMs)<n>They often struggle with complex reasoning due to their limited capacity and a tendency to produce mistakes or inconsistent answers.<n>We introduce ReaLM, a reinforcement learning framework for robust and self-sufficient reasoning in vertical domains.
  arXiv  Detail & Related papers  (2025-08-17T14:50:23Z)
• Reasoning with Exploration: An Entropy Perspective [111.0659496612249]
  Balancing exploration and exploitation is a central goal in reinforcement learning (RL)<n>We introduce a minimal modification to standard RL with only one line of code: augmenting the advantage function with an entropy-based term.<n>Our method achieves significant gains on the Pass@K metric, even when evaluated with extremely large K values.
  arXiv  Detail & Related papers  (2025-06-17T17:54:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.