Reasoning Compression with Mixed-Policy Distillation

• URL: http://arxiv.org/abs/2605.08776v1
• Date: Sat, 09 May 2026 08:04:47 GMT
• Title: Reasoning Compression with Mixed-Policy Distillation
• Abstract summary: Reasoning-centric large language models often incur excessive token usage and high inference-time decoding cost.<n>We propose Mixed-Policy Distillation (MPD), a reasoning compression framework that transfers concise reasoning behavior from a larger-sized teacher to a smaller student.<n>MPD reduces token usage by up to 27.1% while improving performance across multiple reasoning benchmarks.
• Score: 20.7258653384411
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reasoning-centric large language models (LLMs) achieve strong performance by generating intermediate reasoning trajectories, but often incur excessive token usage and high inference-time decoding cost. We observe that, when solving the same problems, larger reasoning models can often produce more concise traces, whereas smaller reasoning models tend to generate longer and more redundant trajectories. This is especially problematic in real-world deployment, where memory, latency, and serving-cost constraints often favor smaller models. Our observations suggest that reasoning compression can be transferred from large models to small ones rather than enforced through explicit length constraints. Based on this insight, we propose Mixed-Policy Distillation (MPD), a reasoning compression framework that transfers concise reasoning behavior from a larger-sized teacher to a smaller student by distilling teacher-compressed student trajectories. Unlike on-policy distillation, which aligns the student with teacher distributions over verbose student trajectories, or off-policy distillation, which relies on teacher-generated trajectories and may suffer from distribution mismatch, MPD combines the strengths of both. Given a student-sampled trajectory, the teacher rewrites it into a more concise reasoning trace, and the student is trained via KL-based alignment on the compressed trajectory. This preserves student-policy exploration while injecting teacher-guided compression. Experiments on Qwen3-1.7B show that MPD reduces token usage by up to 27.1% while improving performance across multiple reasoning benchmarks, demonstrating an effective approach to efficient small-model reasoning.

Related papers

• SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning [54.402550664505576]
  On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models to short-context students.<n>We study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models.<n> Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning.
  arXiv  Detail & Related papers  (2026-08-14T12:57:02Z)
• Contrastive On-Policy Distillation [36.47487086937924]
  We propose COPD, a contrastive On-policy Distillation framework.<n> COPD directly encourages the student model to learn more concise and efficient reasoning strategies.
  arXiv  Detail & Related papers  (2026-07-21T12:35:04Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation [49.98710755440242]
  Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities.<n>We reveal a critical inefficiency in this approach: up to 69% of the prompts in standard chart / document reasoning datasets are effectively zero-delta.<n>We propose a staged synthesis pipeline that repurposes existing datasets as seeds, actively targeting student failure modes to produce better prompts.
  arXiv  Detail & Related papers  (2026-05-15T02:04:12Z)
• Structural Rationale Distillation via Reasoning Space Compression [34.91106623292321]
  Reasoning Path Compression constrains the teacher to follow a compact, dynamically maintained bank of reusable high-level reasoning paths.<n>For each training question, D-RPC retrieves the most relevant path and conditions the teacher to follow it, producing rationales consistent across similar problems yet diverse enough to cover different problem types.
  arXiv  Detail & Related papers  (2026-05-08T02:15:52Z)
• Learning from Partial Chain-of-Thought via Truncated-Reasoning Self-Distillation [15.338493974763471]
  We introduce Truncated-Reasoning Self-Distillation (TRSD), a lightweight post-training procedure that encourages models to produce correct predictions from partial reasoning traces.<n>We demonstrate that TRSD improves robustness to truncated inference, with far reduced accuracy tradeoffs when applied to a diverse set of reasoning models.
  arXiv  Detail & Related papers  (2026-02-27T06:08:22Z)
• ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure [6.494457510860719]
  We identify a reproducible inference-time phenomenon, termed Self-Compression.<n>When multiple independent and answerable questions are presented within a single prompt, the model spontaneously produces shorter reasoning traces for each question.<n>We propose ConPress, a lightweight self-supervised fine-tuning approach.
  arXiv  Detail & Related papers  (2026-02-01T22:31:19Z)
• Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment [82.00769536768509]
  Rank-Surprisal Ratio is a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory.<n>We demonstrate its practical utility in both trajectory selection and teacher selection.
  arXiv  Detail & Related papers  (2026-01-20T18:58:10Z)
• Long-Chain Reasoning Distillation via Adaptive Prefix Alignment [57.130176131042965]
  We propose a framework that exploits teacher CoTs for distillation through adaptive prefix alignment.<n>P-ALIGN adaptively truncates teacher-generated reasoning trajectories by determining whether the remaining suffix is concise.<n>Experiments on multiple mathematical reasoning benchmarks demonstrate that P-ALIGN outperforms all baselines by over 3%.
  arXiv  Detail & Related papers  (2026-01-15T04:40:45Z)
• In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners [12.995634497832027]
  Transferring reasoning capabilities from larger language models to smaller ones often fails counterintuitively.<n>We identify that this failure stems from distributional misalignment: reasoning traces from larger models contain tokens that are low probability under the student's distribution.<n>We propose Reverse Speculative Decoding (RSD), a mechanism for generating student-friendly reasoning traces.
  arXiv  Detail & Related papers  (2025-09-26T11:40:32Z)
• Distilling the Implicit Multi-Branch Structure in LLMs' Reasoning via Reinforcement Learning [63.888013006686364]
  Distilling reasoning paths from teacher to student models via supervised fine-tuning (SFT) provides a shortcut for improving the reasoning ability of Large Language Models (LLMs)<n>We propose RLKD, a reinforcement learning-based distillation framework guided by a novel Generative Structure Reward Model (GSRM)<n>Our GSRM converts reasoning paths into multiple meta-reasoning-solving steps and computes rewards to measure structural alignment between student and teacher reasoning.
  arXiv  Detail & Related papers  (2025-05-22T02:36:36Z)
• SCOTT: Self-Consistent Chain-of-Thought Distillation [68.40232422158569]
  Large language models (LMs) generate free-text rationales for their predictions via chain-of-thought prompting. We propose a faithful knowledge distillation method to learn a small, self-consistent CoT model from a teacher model that is orders of magnitude larger. To ensure faithful distillation, we use the teacher-generated rationales to learn a student LM with a counterfactual reasoning objective.
  arXiv  Detail & Related papers  (2023-05-03T03:47:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.