ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression

• URL: http://arxiv.org/abs/2605.07501v1
• Date: Fri, 08 May 2026 09:37:30 GMT
• Title: ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression
• Abstract summary: Large reasoning models (LRMs) achieve strong performance via extended chain-of-thought (CoT) reasoning, yet suffer from excessive token consumption and high inference latency.<n>We propose textbfExpThinkxspace, an RL framework that addresses both dimensions through two complementary mechanisms.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that textbfExpThinkxspace reduces average response length by up to 77% while simultaneously improving accuracy.
• Score: 14.830831997228657
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large reasoning models (LRMs) achieve strong performance via extended chain-of-thought (CoT) reasoning, yet suffer from excessive token consumption and high inference latency. Existing reinforcement learning (RL) approaches for CoT compression rely on uniform, static length penalties that neglect model capability dynamics and problem-level difficulty variation. We propose \textbf{ExpThink}\xspace, an RL framework that addresses both dimensions through two complementary mechanisms. First, \emph{experience-guided reward shaping} tracks the shortest correct solution found so far for each problem and applies a three-tier reward: full credit for concise correct responses, discounted credit for verbose correct ones, and zero for incorrect ones. The threshold tightens automatically with model improvement, forming a self-evolving curriculum that requires no manual scheduling. Second, \emph{difficulty-adaptive advantage} replaces standard deviation normalization with correct-count normalization, yielding monotonically difficulty-scaled gradients that amplify learning on hard problems to preserve accuracy while suppressing gradients on easy ones to encourage brevity. Together, these mechanisms enforce an accuracy-first, compression-second training objective. Experiments on multiple mathematical reasoning benchmarks demonstrate that \textbf{ExpThink}\xspace reduces average response length by up to 77\% while simultaneously improving accuracy, achieving up to $3\times$ higher accuracy-efficiency ratio (accuracy divided by average token count) than the vanilla baseline and outperforming existing RL-based compression methods on both metrics.

Related papers

• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• SuCo: Sufficiency-guided Continuous Adaptive Reasoning [59.17222918775686]
  Large Reasoning Models (LRMs) often generate excessively long Chain-of-Thoughts (CoT)<n>Existing efforts to mitigate this inefficiency typically rely on discrete reasoning modes or fixed budget tiers.<n>We introduce Minimal Sufficient CoT (MSC), defined as the shortest prefix of a CoT trajectory which is adequate for producing the correct answer.
  arXiv  Detail & Related papers  (2026-06-16T08:52:15Z)
• LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models [24.128148876541783]
  Large reasoning models, such as OpenAI o1 and DeepSeek-R1, tend to become increasingly verbose as their reasoning capabilities improve.<n>These inflated Chain-of-Length trajectories (CoT) often exceed what the underlying problems require, wasting compute, latency, and context budgets.<n>We propose LEAD (Thought-Efficient Adaptive and Dynamic reasoning), a method that replaces statics with online, self-adaptive mechanisms.
  arXiv  Detail & Related papers  (2026-05-10T23:05:02Z)
• SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoning [34.14269693187003]
  We propose SmartThinker, a novel efficient reasoning method with progressive CoT length calibration.<n>Experiment results show that SmartThinker achieves up to 52.5% average length compression with improved accuracy.
  arXiv  Detail & Related papers  (2026-03-09T06:08:14Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• Training-free Context-adaptive Attention for Efficient Long Context Modeling [57.703159205740185]
  Training-free Context-adaptive Attention (TCA-Attention) is a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference.<n>TCA-Attention achieves a 2.8$times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention.
  arXiv  Detail & Related papers  (2025-12-10T01:54:57Z)
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains [56.708381920156256]
  Large Reasoning Models (LRMs) have demonstrated impressive capabilities but suffer from cognitive inefficiencies like overthinking'' simple problems and underthinking'' complex ones.<n>This paper introduces textbfDeepCompress, a novel framework that simultaneously enhances both the accuracy and efficiency of LRMs.
  arXiv  Detail & Related papers  (2025-10-31T12:13:11Z)
• DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning [134.03095505580276]
  Doing Length pEnalty Right (DLER) is a training recipe combining batch-wise reward normalization, higher clipping, dynamic sampling, and a simple truncation length penalty.<n>DLER achieves state-of-the-art accuracy--efficiency trade-offs, cutting output length by over 70 percent while surpassing all previous baseline accuracy.
  arXiv  Detail & Related papers  (2025-10-16T20:05:57Z)
• SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression [48.04180854972225]
  We introduce SIRI, Scaling Iterative Reinforcement Learning with Interleaved Compression, a simple yet effective RL approach for Large Reasoning Models (LRMs)<n>We show that this trade-off can be overcome through a training regime that iteratively alternates between compressing and expanding the reasoning budget.<n>Remarkably, we find that after each compression-expansion cycle, the model's performance improves even as its output length decreases.
  arXiv  Detail & Related papers  (2025-09-29T17:59:08Z)
• R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning [80.104336426172]
  Chain-of-thought (CoT) enhances problem-solving ability of large language models.<n>CoT incurs substantial inference cost due to long autoregressive trajectories.<n>We introduce R-Stitch, a training-free hybrid decoding framework.
  arXiv  Detail & Related papers  (2025-07-23T08:14:36Z)
• Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards [17.829990749622496]
  We propose an adaptive reward-shaping method for large language models.<n>Our method dynamically adjusts the trade-off between accuracy and response length based on model performance.<n> Experiments show that our approach consistently and dramatically reduces reasoning length while largely maintaining accuracy.
  arXiv  Detail & Related papers  (2025-05-23T18:44:46Z)
• Accelerating Chain-of-Thought Reasoning: When Goal-Gradient Importance Meets Dynamic Skipping [3.521097198612099]
  Adaptive GoGI-Skip is a novel framework learning dynamic CoT compression via supervised fine-tuning.<n>It achieves substantial efficiency gains - reducing CoT token counts by over 45% on average and delivering 1.6-2.0 times inference speedups.<n> Notably, it significantly outperforms existing baselines by preserving accuracy even at high effective compression rates.
  arXiv  Detail & Related papers  (2025-05-13T09:39:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.