Learning to Reason Efficiently with Discounted Reinforcement Learning

• URL: http://arxiv.org/abs/2510.23486v1
• Date: Mon, 27 Oct 2025 16:17:45 GMT
• Title: Learning to Reason Efficiently with Discounted Reinforcement Learning
• Abstract summary: Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency.<n>We challenge the assumption that longer responses improve accuracy.<n>By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning.
• Score: 57.28725125116862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency. We challenge the assumption that longer responses improve accuracy. By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning. Experiments confirm our theoretical results that this approach shortens chains of thought while preserving accuracy.

Related papers

• ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning [69.64972562984882]
  We propose ThoughtFold, a framework that leverages fine-grained preference learning to redundant explorations for efficient reasoning.<n>We show that ThoughtFold significantly enhances efficiency.<n>It reduces the token usage of DeepSeek-R1-Distill-Qwen-7B by approximately 56% while maintaining state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2026-06-02T11:21:27Z)
• Thinking with Reasoning Skills: Fewer Tokens, More Accuracy [10.06085033727538]
  Reasoning LLMs often spend substantial tokens on long intermediate reasoning traces when solving new problems.<n>We propose to summarize and store reusable reasoning skills distilled from extensive deliberation and trial-and-error exploration.
  arXiv  Detail & Related papers  (2026-04-23T15:12:58Z)
• Reasoning as Compression: Unifying Budget Forcing via the Conditional Information Bottleneck [12.360124156284305]
  Existing "Budget Forcing" methods suppress both essential reasoning and redundant filler.<n>We recast efficient reasoning as a lossy compression problem under the Information Bottleneck (IB) principle.<n>In contrast to naive token-counting-based approaches, we introduce a semantic prior that measures token cost by surprisal under a language model prior.
  arXiv  Detail & Related papers  (2026-03-09T14:56:57Z)
• IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoning [47.55414301744048]
  We argue that existing sequence-level reward-shaping methods offer limited control over how reasoning effort is allocated across tokens.<n>We propose IAPO, an information-theoretic post-training framework that assigns token-wise advantages based on each token's conditional mutual information.<n>IAPO consistently improves reasoning accuracy while reducing reasoning length by up to 36%, outperforming existing token-efficient RL methods.
  arXiv  Detail & Related papers  (2026-02-22T05:30:14Z)
• TokenSqueeze: Performance-Preserving Compression for Reasoning LLMs [57.217593337454026]
  TokenSqueeze is a novel Long2Short method that condenses reasoning paths while preserving performance and relying exclusively on self-generated data.<n>We show that TokenSqueeze reduces token usage while maintaining accuracy on the MATH500 benchmark.
  arXiv  Detail & Related papers  (2025-11-17T10:38:56Z)
• In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback [38.915062716409686]
  InTRO is a new framework that enables both token-level exploration and self-feedback for accurate and concise reasoning.<n>InTRO consistently outperforms other baselines, raising solution accuracy by up to 20% relative to the base model.<n>Its chains of thought are notably more concise, exhibiting reduced verbosity.
  arXiv  Detail & Related papers  (2025-11-13T01:47:06Z)
• DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning [134.03095505580276]
  Doing Length pEnalty Right (DLER) is a training recipe combining batch-wise reward normalization, higher clipping, dynamic sampling, and a simple truncation length penalty.<n>DLER achieves state-of-the-art accuracy--efficiency trade-offs, cutting output length by over 70 percent while surpassing all previous baseline accuracy.
  arXiv  Detail & Related papers  (2025-10-16T20:05:57Z)
• Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models [26.88030285500965]
  Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks but often suffer from excessive verbosity, known as "overthinking"<n>We introduce textbfStep Pruner (SP), an RL framework that steers LRMs toward more efficient reasoning by favoring compact reasoning steps.<n>Our step-aware reward function prioritizes correctness while imposing penalties for redundant steps, and withholds rewards for incorrect responses to prevent the reinforcement of erroneous reasoning.
  arXiv  Detail & Related papers  (2025-10-04T13:24:26Z)
• Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking [50.97239453902612]
  Large Reasoning Models (LRMs) have achieved impressive performance on challenging tasks, yet their deep reasoning often incurs substantial computational costs.<n>Inspired by Evidence Accumulation Models, we find that LRMs have accumulated sufficient information early in reasoning, making further reasoning steps redundant.<n>We propose Just-Enough Thinking (JET), which trains models to proactively terminate unnecessary reasoning.
  arXiv  Detail & Related papers  (2025-09-27T16:25:06Z)
• Train Long, Think Short: Curriculum Learning for Efficient Reasoning [51.506559652495476]
  We propose a curriculum learning strategy for length-controlled reasoning.<n>Our method starts with generous token budgets and gradually tightens them over training.<n>Experiments on GSM8K, MATH500, SVAMP, College Math, and GSM+ demonstrate that curriculum-based training consistently outperforms fixed-budget baselines.
  arXiv  Detail & Related papers  (2025-08-12T13:48:03Z)
• Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards [17.829990749622496]
  We propose an adaptive reward-shaping method for large language models.<n>Our method dynamically adjusts the trade-off between accuracy and response length based on model performance.<n> Experiments show that our approach consistently and dramatically reduces reasoning length while largely maintaining accuracy.
  arXiv  Detail & Related papers  (2025-05-23T18:44:46Z)
• VeriThinker: Learning to Verify Makes Reasoning Model Efficient [52.74493506816969]
  Large Reasoning Models excel at complex tasks using Chain-of-Thought (CoT) reasoning.<n>Their tendency to overthinking leads to unnecessarily lengthy reasoning chains.<n>We introduce VeriThinker, a novel approach for CoT compression.
  arXiv  Detail & Related papers  (2025-05-23T14:17:56Z)
• Concise Reasoning via Reinforcement Learning [13.657506042120167]
  We revisit the core principles of reinforcement learning (RL)<n>We uncover a natural correlation between conciseness and accuracy that has been largely overlooked.<n>We show that introducing a secondary phase of RL training, using a very small set of problems, can significantly reduce chains of thought.
  arXiv  Detail & Related papers  (2025-04-07T15:35:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.