The Overthinker's DIET: Cutting Token Calories with DIfficulty-AwarE Training

• URL: http://arxiv.org/abs/2505.19217v1
• Date: Sun, 25 May 2025 16:24:12 GMT
• Title: The Overthinker's DIET: Cutting Token Calories with DIfficulty-AwarE Training
• Abstract summary: Recent large language models (LLMs) exhibit impressive reasoning but often over-think, generating excessively long responses that hinder efficiency.<n>We introduce DIET, a framework that systematically cuts these "token calories" by integrating on-the-fly problem difficulty into the reinforcement learning process.<n> DIET dynamically adapts token compression strategies by modulating token penalty strength and conditioning target lengths on estimated task difficulty.
• Score: 63.99981166397423
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent large language models (LLMs) exhibit impressive reasoning but often over-think, generating excessively long responses that hinder efficiency. We introduce DIET ( DIfficulty-AwarE Training), a framework that systematically cuts these "token calories" by integrating on-the-fly problem difficulty into the reinforcement learning (RL) process. DIET dynamically adapts token compression strategies by modulating token penalty strength and conditioning target lengths on estimated task difficulty, to optimize the performance-efficiency trade-off. We also theoretically analyze the pitfalls of naive reward weighting in group-normalized RL algorithms like GRPO, and propose Advantage Weighting technique, which enables stable and effective implementation of these difficulty-aware objectives. Experimental results demonstrate that DIET significantly reduces token counts while simultaneously improving reasoning performance. Beyond raw token reduction, we show two crucial benefits largely overlooked by prior work: (1) DIET leads to superior inference scaling. By maintaining high per-sample quality with fewer tokens, it enables better scaling performance via majority voting with more samples under fixed computational budgets, an area where other methods falter. (2) DIET enhances the natural positive correlation between response length and problem difficulty, ensuring verbosity is appropriately allocated, unlike many existing compression methods that disrupt this relationship. Our analyses provide a principled and effective framework for developing more efficient, practical, and high-performing LLMs.

Related papers

• ADaPT: Token-Level Decoupling for Efficient Large Reasoning Models [48.426487984321675]
  Large reasoning models rely on long chain-of-thought to achieve strong performance, but applying such reasoning uniformly incurs high computational cost.<n>Existing efficiency-oriented methods attempt to shorten or mix reasoning strategies, yet often degrade reasoning capability.<n>We propose Adaptive Dual-Process Thinking (ADaPT), a token-level dual-process framework that explicitly decouples efficiency and correctness signals during training.
  arXiv  Detail & Related papers  (2026-06-18T08:11:45Z)
• Better, Faster: Harnessing Self-Improvement in Large Reasoning Models [88.9107786925265]
  We propose HSIR, which effectively Harnesses Self-Improvement in large Reasoning models via two simple-yet-effective approaches.<n>Specifically, HSIR introduces a verify-then-exit sampling strategy to mitigate data imbalance.<n>HSIR also designs an Intrinsic Diversity score to quantify overthinking and filter out the undesired solutions.
  arXiv  Detail & Related papers  (2026-05-24T10:54:46Z)
• LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models [24.128148876541783]
  Large reasoning models, such as OpenAI o1 and DeepSeek-R1, tend to become increasingly verbose as their reasoning capabilities improve.<n>These inflated Chain-of-Length trajectories (CoT) often exceed what the underlying problems require, wasting compute, latency, and context budgets.<n>We propose LEAD (Thought-Efficient Adaptive and Dynamic reasoning), a method that replaces statics with online, self-adaptive mechanisms.
  arXiv  Detail & Related papers  (2026-05-10T23:05:02Z)
• Gradually Compacting Large Language Models for Reasoning Like a Boiling Frog [72.4168434368873]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, but their substantial size often demands significant computational resources.<n>We propose a gradual compacting method that divides the compression process into multiple fine-grained iterations.<n>This iterative approach-reminiscent of the "boiling frog" effect-enables the model to be progressively compressed without abrupt performance loss.
  arXiv  Detail & Related papers  (2026-02-04T06:56:52Z)
• Anti-Length Shift: Dynamic Outlier Truncation for Training Efficient Reasoning Models [29.56923793047279]
  We introduce Dynamic Outlier Truncation (DOT), a training-time intervention that selectively suppresses redundant tokens.<n>DOT targets only the extreme tail of response lengths within fully correct rollout groups while preserving long-horizon reasoning capabilities.<n>Our method reduces inference token usage by 78% while simultaneously increasing accuracy compared to the initial policy.
  arXiv  Detail & Related papers  (2026-01-07T14:31:07Z)
• Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model [12.881680088950008]
  Leash is a reinforcement learning framework for efficient reasoning in LLMs.<n>Leash reduces the average reasoning length by 60% across diverse tasks.<n>Our work thus presents a practical and effective paradigm for developing controllable and efficient LLMs.
  arXiv  Detail & Related papers  (2025-12-25T07:16:26Z)
• Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models [50.84995206660551]
  We introduce Conditional advANtage estimatiON (CANON) to amplify the impact of a target metric without presuming its direction.<n>CANON based on entropy consistently outperforms prior methods on both math reasoning and high-complexity logic tasks.
  arXiv  Detail & Related papers  (2025-09-28T16:33:07Z)
• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle [53.239242017802056]
  Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM)<n>However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing and Rollout Silencing.<n>We propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition.
  arXiv  Detail & Related papers  (2025-08-07T17:53:47Z)
• Exploring and Exploiting the Inherent Efficiency within Large Reasoning Models for Self-Guided Efficiency Enhancement [101.77467538102924]
  Large reasoning models (LRMs) exhibit overthinking, which hinders efficiency and inflates inference cost.<n>We propose two lightweight methods to enhance LRM efficiency.<n>First, we introduce Efficiency Steering, a training-free activation steering technique that modulates reasoning behavior via a single direction.<n>Second, we develop Self-Rewarded Efficiency RL, a reinforcement learning framework that dynamically balances task accuracy and brevity.
  arXiv  Detail & Related papers  (2025-06-18T17:18:12Z)
• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping [23.626013831589212]
  Large Reasoning Models (LRMs) have shown remarkable capabilities in solving complex problems through reinforcement learning (RL)<n>We present a unified framework that formulates various efficient reasoning methods through the lens of length-based reward shaping.<n>Experiments on DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, and DeepSeek-R1-Distill-Qwen-32B show that our approach significantly enhances both reasoning performance and response length efficiency.
  arXiv  Detail & Related papers  (2025-05-21T15:03:26Z)
• Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning [66.8042627609456]
  Loss reweighting has shown significant benefits for machine unlearning with large language models (LLMs)<n>In this paper, we identify two distinct goals of loss reweighting, namely, Saturation and Importance.<n>We propose SatImp, a simple reweighting method that combines the advantages of both saturation and importance.
  arXiv  Detail & Related papers  (2025-05-17T10:41:22Z)
• FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing [17.01412432658081]
  Large language models (LLMs) have demonstrated superior performance across various tasks by adhering to scaling laws.<n>We propose a fine-grained token-wise pruning approach for the LLMs, which presents a learnable router to adaptively identify the less important tokens.<n>Our approach achieves state-of-the-art (SOTA) pruning results, surpassing other existing pruning methods.
  arXiv  Detail & Related papers  (2024-12-16T07:09:46Z)
• Balancing LoRA Performance and Efficiency with Simple Shard Sharing [8.827921242078883]
  textbfOptimal textbfShard textbfSharing textbfIntegration in textbfLoRA, a novel PEFT approach that addresses this trade-off through a simple shard-sharing mechanism.<n>Fossils significantly outperforms standard LoRA and its prominent variants in both model performance metrics and computational efficiency.
  arXiv  Detail & Related papers  (2024-09-19T10:26:42Z)
• PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation [61.57833648734164]
  We propose a novel Parallel Yielding Re-Activation (PYRA) method for training-inference efficient task adaptation. PYRA outperforms all competing methods under both low compression rate and high compression rate.
  arXiv  Detail & Related papers  (2024-03-14T09:06:49Z)
• Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
  Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples. We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment. We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
  arXiv  Detail & Related papers  (2022-03-23T06:24:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.