Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training

• URL: http://arxiv.org/abs/2605.07316v1
• Date: Fri, 08 May 2026 06:25:08 GMT
• Title: Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training
• Abstract summary: We propose an on-policy regularization method whose compression signal comes from a virtual shorter distribution induced by the shortest correct responses in rollout groups.<n>Experiments on three reasoning backbones and multiple mathematical and knowledge-intensive benchmarks show that ICR consistently shortens responses while preserving or improving accuracy.
• Score: 11.132427208920424
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning with verifiable rewards improves LLM reasoning but often induces overthinking, where models generate unnecessarily long reasoning traces. Existing methods mainly rely on length penalties or early-exit strategies; however, the former may degrade accuracy and induce underthinking, whereas the latter assumes that substantial portions of reasoning traces can be safely truncated. To obtain a compression signal without these limitations, we revisit the training dynamics of existing compression methods. We observe that the length--accuracy correlation is initially negative but continually increases during compression, indicating that shorter responses are initially more likely to be correct but gradually lose this property as the policy moves toward underthinking. Based on this observation, we formalize overthinking: a negative correlation indicates an overthinking regime, while a positive one indicates underthinking. When overthinking, the shortest correct responses are shorter than the group-average response length in expectation, making them natural compression targets already present in on-policy rollouts. We therefore propose \emph{Implicit Compression Regularization} (ICR), an on-policy regularization method whose compression signal comes from a virtual shorter distribution induced by the shortest correct responses in rollout groups, guiding the policy toward concise yet correct trajectories. Training dynamics show that ICR maintains a better length--accuracy correlation during compression, indicating that short responses remain better aligned with correctness instead of drifting toward underthinking. Experiments on three reasoning backbones and multiple mathematical and knowledge-intensive benchmarks show that ICR consistently shortens responses while preserving or improving accuracy, achieving a stronger accuracy--length Pareto frontier.

Related papers

• ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning [69.64972562984882]
  We propose ThoughtFold, a framework that leverages fine-grained preference learning to redundant explorations for efficient reasoning.<n>We show that ThoughtFold significantly enhances efficiency.<n>It reduces the token usage of DeepSeek-R1-Distill-Qwen-7B by approximately 56% while maintaining state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2026-06-02T11:21:27Z)
• Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains [9.979461886634022]
  Explicit chain-of-thought (CoT) reasoning substantially improves the reasoning ability of large language models.<n>Existing latent reasoning methods treat reasoning as uniformly compressible, causing precision-critical intermediate steps to be overly compressed.<n>We propose Selective Latent Thinking (SLT), a framework that selectively compresses redundant reasoning spans into latent representations.
  arXiv  Detail & Related papers  (2026-05-25T11:57:09Z)
• ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression [14.830831997228657]
  Large reasoning models (LRMs) achieve strong performance via extended chain-of-thought (CoT) reasoning, yet suffer from excessive token consumption and high inference latency.<n>We propose textbfExpThinkxspace, an RL framework that addresses both dimensions through two complementary mechanisms.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that textbfExpThinkxspace reduces average response length by up to 77% while simultaneously improving accuracy.
  arXiv  Detail & Related papers  (2026-05-08T09:37:30Z)
• OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Models [12.17078443900398]
  On-Policy Self-Distillation (OPSD) has recently emerged as an alternative to Reinforcement Learning with Verifiable Rewards (RLVR)<n>We show that in thinking-enabled mathematical reasoning, OPSD behaves most reliably as a compression mechanism rather than a correction mechanism.
  arXiv  Detail & Related papers  (2026-05-07T13:04:34Z)
• Shorter Thoughts, Same Answers: Difficulty-Scaled Segment-Wise RL for CoT Compression [3.6889211112573985]
  Chain-of-thought (CoT) improves reasoning reliability but increases token cost.<n> naive RL-based compression can undesirably shorten the user-facing answer.<n>We propose Difficulty-Scaled Segment-Wise GRPO.
  arXiv  Detail & Related papers  (2026-03-08T11:49:24Z)
• Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning [66.22060690012512]
  Large reasoning models improve with more test-time computation, but often overthink, producing unnecessarily long chains-of-thought that raise cost without improving accuracy.<n>We propose Step-wise Adaptive Penalization (SWAP), a fine-grained framework that allocates length reduction across steps based on intrinsic contribution.
  arXiv  Detail & Related papers  (2026-02-27T20:23:59Z)
• Constraint-Rectified Training for Efficient Chain-of-Thought [60.52883907721588]
  Chain-of-Thought (CoT) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs)<n>While longer reasoning traces can improve answer quality and unlock abilities such as self-correction, they also incur high inference costs and often introduce redundant steps, known as overthinking.<n>Recent research seeks to develop efficient reasoning strategies that balance reasoning length and accuracy.
  arXiv  Detail & Related papers  (2026-02-13T02:13:45Z)
• APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards [61.52322047892064]
  Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs)<n>We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process.<n>We propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST.
  arXiv  Detail & Related papers  (2026-01-31T14:53:20Z)
• Stop Spinning Wheels: Mitigating LLM Overthinking via Mining Patterns for Early Reasoning Exit [114.83867400179354]
  Overthinking can degrade overall performance of large language models.<n>We categorize reasoning into three stages: insufficient exploration stage, compensatory reasoning stage, and reasoning convergence stage.<n>We develop a lightweight thresholding strategy based on rules to improve reasoning accuracy.
  arXiv  Detail & Related papers  (2025-08-25T03:17:17Z)
• ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoning [64.93140713419561]
  Large Reasoning Models (LRMs) perform strongly in complex reasoning tasks via Chain-of-Thought (CoT) prompting, but often suffer from verbose outputs.<n>Existing fine-tuning-based compression methods either operate post-hoc pruning, risking disruption to reasoning coherence, or rely on sampling-based selection.<n>We introduce ConCISE, a framework designed to generate concise reasoning chains, integrating Confidence Injection to boost reasoning confidence, and Early Stopping to terminate reasoning when confidence is sufficient.
  arXiv  Detail & Related papers  (2025-05-08T01:40:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.