Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Model

• URL: http://arxiv.org/abs/2506.23840v1
• Date: Mon, 30 Jun 2025 13:30:33 GMT
• Title: Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Model
• Authors: Bowen Ding, Yuhan Chen, Futing Wang, Lingfeng Ming, Tao Lin,
• Abstract summary: Large Reasoning Models (LRMs) excel at solving complex problems but face an overthinking dilemma.<n>When handling simple tasks, they often produce verbose responses overloaded with thinking tokens.<n>These tokens trigger unnecessary high-level reasoning behaviors like reflection and backtracking, reducing efficiency.
• Score: 7.8354921036790275
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Reasoning Models (LRMs) excel at solving complex problems but face an overthinking dilemma. When handling simple tasks, they often produce verbose responses overloaded with thinking tokens (e.g., wait, however). These tokens trigger unnecessary high-level reasoning behaviors like reflection and backtracking, reducing efficiency. In this work, our pilot study reveals that these thinking-token-induced behaviors are not essential for effective problem-solving and may even hinder correct reasoning within constrained token budgets. We identify this phenomenon as the thinking trap. To mitigate this issue, we propose Dual Policy Preference Optimization (DuP-PO), a novel algorithm featuring: (1) A rollout sampling strategy that guarantees balanced exposure to responses with and without thinking tokens; (2) A fine-grained advantage control technique to dynamically regulate the prediction of target tokens; (3) A policy shaping method ensuring stable gradient contributions from thinking tokens. Experimental results on five popular math reasoning benchmarks show that DuP-PO performs well on the popular LRM, which significantly improves their token efficiency during reasoning, while achieving superior performance of the base model.

Related papers

• IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoning [47.55414301744048]
  We argue that existing sequence-level reward-shaping methods offer limited control over how reasoning effort is allocated across tokens.<n>We propose IAPO, an information-theoretic post-training framework that assigns token-wise advantages based on each token's conditional mutual information.<n>IAPO consistently improves reasoning accuracy while reducing reasoning length by up to 36%, outperforming existing token-efficient RL methods.
  arXiv  Detail & Related papers  (2026-02-22T05:30:14Z)
• Dynamic Thinking-Token Selection for Efficient Reasoning in Large Reasoning Models [29.494777688316674]
  Large Reasoning Models (LRMs) excel at solving complex problems by explicitly generating a reasoning trace before deriving the final answer.<n>However, extended generations incur substantial memory footprint and computational overhead, bottlenecking LRMs' efficiency.<n>This work uses attention maps to analyze the influence of reasoning traces and uncover an interesting phenomenon.<n>Only some decision-critical tokens in a reasoning trace steer the model toward the final answer, while the remaining tokens contribute negligibly.
  arXiv  Detail & Related papers  (2026-01-26T11:31:40Z)
• One Token Embedding Is Enough to Deadlock Your Large Reasoning Model [91.48868589442837]
  We present the Deadlock Attack, a resource exhaustion method that hijacks an LRM's generative control flow.<n>Our method achieves a 100% attack success rate across four advanced LRMs.
  arXiv  Detail & Related papers  (2025-10-12T07:42:57Z)
• Entropy After $\langle \ exttt{/Think} \ angle$ for reasoning model early exiting [38.93424884988798]
  We show that large reasoning models overthink, continuing to revise answers even after reaching the correct solution.<n>We propose Entropy After /Think> (EAT) for monitoring and deciding whether to exit reasoning early.<n>EAT reduces token usage by 13 - 21% without harming accuracy.
  arXiv  Detail & Related papers  (2025-09-30T16:59:37Z)
• Promoting Efficient Reasoning with Verifiable Stepwise Reward [7.385337642642193]
  Large reasoning models (LRMs) have recently achieved significant progress in complex reasoning tasks, aided by reinforcement learning.<n>LRMs often suffer from overthinking, expending excessive computation on simple problems and reducing efficiency.<n>We propose a novel rule-based verifiable stepwise reward mechanism (VSRM), which assigns rewards based on the performance of intermediate states in the reasoning trajectory.
  arXiv  Detail & Related papers  (2025-08-14T02:43:53Z)
• Is Long-to-Short a Free Lunch? Investigating Inconsistency and Reasoning Efficiency in LRMs [8.359909829007005]
  We investigate whether efficient reasoning strategies introduce behavioral inconsistencies in large reasoning models (LRMs)<n>$ICBENCH$ is a benchmark designed to measure inconsistency in LRMs across three dimensions.<n>We find that while larger models generally exhibit greater consistency than smaller ones, they all display widespread "scheming" behaviors.
  arXiv  Detail & Related papers  (2025-06-24T10:25:28Z)
• Multipole Attention for Efficient Long Context Reasoning [64.94673641704289]
  Large Reasoning Models (LRMs) have shown promising accuracy improvements on complex problem-solving tasks.<n>LRMs need to generate long chain-of-thought reasoning in order to think before answering.<n>We introduce Multipole Attention, which accelerates autoregressive reasoning by only computing exact attention for the most important tokens.
  arXiv  Detail & Related papers  (2025-06-16T03:00:40Z)
• On Reasoning Strength Planning in Large Reasoning Models [50.61816666920207]
  We find evidence that LRMs pre-plan the reasoning strengths in their activations even before generation.<n>We then uncover that LRMs encode this reasoning strength through a pre-allocated directional vector embedded in the activations of the model.<n>Our work provides new insights into the internal mechanisms of reasoning in LRMs and offers practical tools for controlling their reasoning behaviors.
  arXiv  Detail & Related papers  (2025-06-10T02:55:13Z)
• Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiency [24.56015832583054]
  Explicit self-reflection, signaled by tokens such as "Wait" and "Hmm", is necessary for advanced reasoning.<n>We propose NoWait, a simple yet effective approach that disables explicit self-reflection by suppressing these tokens during inference.
  arXiv  Detail & Related papers  (2025-06-10T01:54:04Z)
• CoThink: Token-Efficient Reasoning via Instruct Models Guiding Reasoning Models [56.40065909544213]
  Large language models (LLMs) benefit from increased test-time compute, a phenomenon known as test-time scaling.<n>However, reasoning-optimized models often overthink even simple problems, producing excessively verbose outputs and leading to low token efficiency.<n>We identify two key causes of this verbosity: (1) reinforcement learning reduces the information density of forward reasoning, and (2) backward chain-of thought training encourages redundant and often unnecessary verification steps.
  arXiv  Detail & Related papers  (2025-05-28T06:24:45Z)
• Efficient Reasoning via Chain of Unconscious Thought [40.82356218832031]
  Large Reasoning Models (LRMs) achieve promising performance but compromise token efficiency due to verbose reasoning processes.<n>We propose a new reasoning paradigm, termed Chain of Unconscious Thought (CoUT), to improve the token efficiency of LRMs.<n>Our work reveals that models may possess beneficial unconscious thought, enabling improved efficiency without sacrificing performance.
  arXiv  Detail & Related papers  (2025-05-26T09:34:04Z)
• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models [67.87579664988199]
  TON is a two-stage training strategy for vision-language models (VLMs)<n>It introduces a think-or-not format that serves as a cold start for selective reasoning.<n>TON can reduce the completion length by up to 90% compared to vanilla GRPO.
  arXiv  Detail & Related papers  (2025-05-22T16:13:29Z)
• Let LLMs Break Free from Overthinking via Self-Braking Tuning [60.08396797526657]
  Large reasoning models (LRMs) have significantly enhanced their reasoning capabilities by generating longer chains of thought.<n>This performance gain comes at the cost of a substantial increase in redundant reasoning during the generation process.<n>We propose a novel framework, Self-Braking Tuning (SBT), which tackles overthinking from the perspective of allowing the model to regulate its own reasoning process.
  arXiv  Detail & Related papers  (2025-05-20T16:53:40Z)
• Efficient Inference for Large Reasoning Models: A Survey [41.239535590590435]
  Large Reasoning Models (LRMs) significantly improve the reasoning ability of Large Language Models (LLMs) by learning to reason.<n>However, their deliberative reasoning process leads to inefficiencies in token usage, memory consumption, and inference time.<n>This survey provides a review of efficient inference methods designed specifically for LRMs, focusing on mitigating token inefficiency while preserving the reasoning quality.
  arXiv  Detail & Related papers  (2025-03-29T13:27:46Z)
• Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability [53.51560766150442]
  Critical tokens are elements within reasoning trajectories that significantly influence incorrect outcomes.<n>We present a novel framework for identifying these tokens through rollout sampling.<n>We show that identifying and replacing critical tokens significantly improves model accuracy.
  arXiv  Detail & Related papers  (2024-11-29T18:58:22Z)
• Self-Evaluation Guided Beam Search for Reasoning [61.523627290397556]
  We introduce a stepwise self-evaluation mechanism to guide and calibrate the reasoning process of Large Language Model (LLM) We propose a decoding algorithm integrating the self-evaluation guidance via beam search. Our approach surpasses the corresponding Codex-backboned baselines in few-shot accuracy by $6.34%$, $9.56%$, and $5.46%$ on the GSM8K, AQuA, and StrategyQA.
  arXiv  Detail & Related papers  (2023-05-01T02:37:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.