Reasoning as Compression: Unifying Budget Forcing via the Conditional Information Bottleneck

• URL: http://arxiv.org/abs/2603.08462v1
• Date: Mon, 09 Mar 2026 14:56:57 GMT
• Title: Reasoning as Compression: Unifying Budget Forcing via the Conditional Information Bottleneck
• Abstract summary: Existing "Budget Forcing" methods suppress both essential reasoning and redundant filler.<n>We recast efficient reasoning as a lossy compression problem under the Information Bottleneck (IB) principle.<n>In contrast to naive token-counting-based approaches, we introduce a semantic prior that measures token cost by surprisal under a language model prior.
• Score: 12.360124156284305
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chain-of-Thought (CoT) prompting improves LLM accuracy on complex tasks but often increases token usage and inference cost. Existing "Budget Forcing" methods reducing cost via fine-tuning with heuristic length penalties, suppress both essential reasoning and redundant filler. We recast efficient reasoning as a lossy compression problem under the Information Bottleneck (IB) principle, and identify a key theoretical gap when applying naive IB to transformers: attention violates the Markov property between prompt, reasoning trace, and response. To resolve this issue, we model CoT generation under the Conditional Information Bottleneck (CIB) principle, where the reasoning trace Z acts as a computational bridge that contains only the information about the response Y that is not directly accessible from the prompt X. This yields a general Reinforcement Learning objective: maximize task reward while compressing completions under a prior over reasoning traces, subsuming common heuristics (e.g., length penalties) as special cases (e.g., uniform priors). In contrast to naive token-counting-based approaches, we introduce a semantic prior that measures token cost by surprisal under a language model prior. Empirically, our CIB objective prunes cognitive bloat while preserving fluency and logic, improving accuracy at moderate compression and enabling aggressive compression with minimal accuracy drop.

Related papers

• Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models [62.522141464687195]
  We introduce Action-JND, which extends the principle of just noticeable difference (JND) to embodied perception.<n>A token change is considered tolerable only when the induced action deviation remains within a tolerated margin.<n>The resulting action-tolerance score serves as a plug-and-play criterion for VLA compression paradigms.
  arXiv  Detail & Related papers  (2026-08-21T15:59:37Z)
• Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs [36.73612668202756]
  HAB learns step-specific token budgeting signals from PPL-derived step comparisons.<n>Experiments on GSM8K and MATH500 show that HAB not only surpasses standard CoT in accuracy but also reduces token usage.
  arXiv  Detail & Related papers  (2026-05-31T11:20:00Z)
• Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoning [39.72119774004103]
  Chain-of-Thought (CoT) has substantially empowered Large Language Models (LLMs) to tackle complex reasoning tasks.<n>The verbose nature of explicit reasoning steps incurs prohibitive inference latency and computational costs, limiting real-world deployment.<n>We propose Compress responses for Easy questions and Explore Hard ones (CEEH), a difficulty-aware approach to RL-based efficient reasoning.
  arXiv  Detail & Related papers  (2026-02-26T05:47:30Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards [61.52322047892064]
  Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs)<n>We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process.<n>We propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST.
  arXiv  Detail & Related papers  (2026-01-31T14:53:20Z)
• ReGuLaR: Variational Latent Reasoning Guided by Rendered Chain-of-Thought [49.203970812338916]
  Explicit reasoning chains introduce substantial computational redundancy.<n>Recent latent reasoning methods attempt to mitigate this by compressing reasoning processes into latent space.<n>We propose Rendered CoT-Guided variational Latent Reasoning (ReGuLaR)
  arXiv  Detail & Related papers  (2026-01-30T17:08:06Z)
• TokenSqueeze: Performance-Preserving Compression for Reasoning LLMs [57.217593337454026]
  TokenSqueeze is a novel Long2Short method that condenses reasoning paths while preserving performance and relying exclusively on self-generated data.<n>We show that TokenSqueeze reduces token usage while maintaining accuracy on the MATH500 benchmark.
  arXiv  Detail & Related papers  (2025-11-17T10:38:56Z)
• R-Capsule: Compressing High-Level Plans for Efficient Large Language Model Reasoning [25.87953249848607]
  Chain-of-Thought (CoT) prompting helps Large Language Models (LLMs) tackle complex reasoning by eliciting explicit step-by-step rationales.<n>We propose the Reasoning Capsule (R-Capsule), a framework that aims to combine the efficiency of latent reasoning with the transparency of explicit CoT.
  arXiv  Detail & Related papers  (2025-09-26T09:53:41Z)
• Fractured Chain-of-Thought Reasoning [61.647243580650446]
  We introduce Fractured Sampling, a unified inference-time strategy that interpolates between full CoT and solution-only sampling.<n>We show that Fractured Sampling consistently achieves superior accuracy-cost trade-offs, yielding steep log-linear scaling gains in Pass@k versus token budget.
  arXiv  Detail & Related papers  (2025-05-19T11:30:41Z)
• SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning [14.020244011380063]
  SpecReason is a system that accelerates LRM inference.<n>It exploits the semantic flexibility of thinking tokens in preserving final-answer accuracy.<n>It achieves $1.4-3.0times$ speedup over vanilla LRM inference.
  arXiv  Detail & Related papers  (2025-04-10T16:05:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.