Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models

• URL: http://arxiv.org/abs/2607.28707v1
• Date: Thu, 30 Jul 2026 15:59:51 GMT
• Title: Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models
• Abstract summary: Entropy-based pruning has been proposed as an effective method for compressing Chain-of-Thought (CoT) reasoning with negligible accuracy loss.<n>We show that entropy offers no advantage over random pruning in any evaluated setting.
• Score: 16.372524974789492
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Entropy-based pruning has been proposed as an effective method for compressing Chain-of-Thought (CoT) reasoning with negligible accuracy loss. We test the robustness of low- and high-entropy CoT step selection methods across various models and reasoning tasks, showing that entropy offers no advantage over random pruning in any evaluated setting. Moving from sentences to tokens, we then show that retaining low-entropy tokens seems effective only on mathematical benchmarks. We find this is due to the inherently low-entropy nature of numeric tokens, which also convey semantic content in such problems. Finally, we demonstrate that patching a subset of a few CoT tokens with their original activations recovers near-perfect full-trace performance, providing causal evidence that task information is not concentrated in a small set of CoT tokens identifiable by heuristics, but rather distributed across the full reasoning chain.

Related papers

• Revisiting Chain-of-Thought Reasoning under Limited Supervision: Semi-supervised Chain-of-Thought Learning [17.893249958828218]
  We propose a framework that uses unlabeled questions to construct pseudo reasoning supervision.<n>Semi-CoT samples multiple pseudo-CoTs for each unlabeled question, estimates answer-level semantic entropy, and selects low-entropy reasoning chains as reliable pseudo-CoT demonstrations.<n>Pilot experiments on AQuA, SVAMP, GSM8K, and MultiArith show that the entropy gate selects high-precision pseudo-CoTs, with pseudo-answer precision ranging from $91.36%$ to $100%$.
  arXiv  Detail & Related papers  (2026-07-01T22:17:39Z)
• Reasoning Quality Emerges Early: Data Curation for Reasoning Models [58.56882815783977]
  We show that diverse and challenging reasoning examples can be identified using only the initial reasoning tokens.<n>Our method outperforms existing baselines by up to 1.7% while being 91% more token efficient.
  arXiv  Detail & Related papers  (2026-06-25T09:32:58Z)
• Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens [4.0704009036918025]
  Reasoning in Large Language Models incurs significant inference-time compute.<n>We show that reasoning tokens split into two functional types: low-entropy textitstructural tokens (recurring phrases that scaffold the reasoning process) and higher-entropy textitorganic tokens (problem-specific content that drives toward a solution)<n>We derive textitsupertokens that capture frequent structural patterns, then teach the model to adopt them via supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-04-29T07:06:43Z)
• When Chain-of-Thought Fails, the Solution Hides in the Hidden States [6.13796568285414]
  We use activation patching to transfer token-level hidden states from a CoT generation to a direct-answer run for the same question.<n>We find that individual CoT tokens can encode sufficient information to recover the correct answer, even when the original trace is incorrect.
  arXiv  Detail & Related papers  (2026-04-25T15:33:21Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimization [78.94590726578014]
  multimodal reasoning models (MLRMs) remain prone to hallucinations, and effective solutions are still underexplored.<n>We propose C3PO, a training-based mitigation framework comprising textbfCompression and textbfPreference textbfOptimization.
  arXiv  Detail & Related papers  (2026-02-03T11:00:55Z)
• EntroCoT: Enhancing Chain-of-Thought via Adaptive Entropy-Guided Segmentation [18.606842425858]
  Chain-of-Thought (CoT) prompting has significantly enhanced the mathematical reasoning capabilities of Large Language Models.<n>Existing fine-tuning datasets frequently suffer from the "answer right but reasoning wrong" probelm.<n>This paper proposes EntroCoT, a unified framework for automatically identifying and refining low-quality CoT supervision traces.
  arXiv  Detail & Related papers  (2026-01-07T10:02:27Z)
• Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought [16.907732581097417]
  We focus on Chain-of-Continuous-Thought (COCONUT) which claims better efficiency and stability than explicit Chain-of-Thought (CoT)<n>Unlike CoT tokens, COCONUT tokens show minimal sensitivity to steering and lack reasoning-critical information.<n>Results on MMLU and HotpotQA demonstrate that COCONUT consistently exploits dataset artifacts, inflating benchmark performance without true reasoning.
  arXiv  Detail & Related papers  (2025-12-25T15:14:53Z)
• Fractured Chain-of-Thought Reasoning [61.647243580650446]
  We introduce Fractured Sampling, a unified inference-time strategy that interpolates between full CoT and solution-only sampling.<n>We show that Fractured Sampling consistently achieves superior accuracy-cost trade-offs, yielding steep log-linear scaling gains in Pass@k versus token budget.
  arXiv  Detail & Related papers  (2025-05-19T11:30:41Z)
• CoT-Valve: Length-Compressible Chain-of-Thought Tuning [50.196317781229496]
  We introduce a new tuning and inference strategy named CoT-Valve, designed to allow models to generate reasoning chains of varying lengths.<n>We show that CoT-Valve successfully enables controllability and compressibility of the chain and shows better performance than the prompt-based control.
  arXiv  Detail & Related papers  (2025-02-13T18:52:36Z)
• Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods [59.779795063072655]
  Chain-of-Thought (CoT) prompting and its variants have gained popularity as effective methods for solving multi-step reasoning problems. We analyze CoT prompting from a statistical estimation perspective, providing a comprehensive characterization of its sample complexity.
  arXiv  Detail & Related papers  (2024-08-25T04:07:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.