Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization

• URL: http://arxiv.org/abs/2501.17974v2
• Date: Fri, 31 Jan 2025 16:06:26 GMT
• Title: Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization
• Abstract summary: In this work, we propose a way to allow models to be aware of inference budgets by formulating it as utility with respect to an inference budget constraint.<n>In a nutshell, models fine-tuned through IBPO learn to understand'' the difficulty of queries and allocate inference budgets to harder ones.<n>These improvements are approximately $2$x those of self-consistency under the same budgets.
• Score: 48.5702580429946
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Solving mathematics problems has been an intriguing capability of large language models, and many efforts have been made to improve reasoning by extending reasoning length, such as through self-correction and extensive long chain-of-thoughts. While promising in problem-solving, advanced long reasoning chain models exhibit an undesired single-modal behavior, where trivial questions require unnecessarily tedious long chains of thought. In this work, we propose a way to allow models to be aware of inference budgets by formulating it as utility maximization with respect to an inference budget constraint, hence naming our algorithm Inference Budget-Constrained Policy Optimization (IBPO). In a nutshell, models fine-tuned through IBPO learn to ``understand'' the difficulty of queries and allocate inference budgets to harder ones. With different inference budgets, our best models are able to have a $4.14$\% and $5.74$\% absolute improvement ($8.08$\% and $11.2$\% relative improvement) on MATH500 using $2.16$x and $4.32$x inference budgets respectively, relative to LLaMA3.1 8B Instruct. These improvements are approximately $2$x those of self-consistency under the same budgets.

Related papers

• Nice Fold or Hero Call: Learning Budget-Efficient Thinking for Adaptive Reasoning [9.687158786222994]
  Large reasoning models (LRMs) improve problem solving through extended reasoning, but often misallocate test-time compute.<n>We propose Budget-Efficient Thinking (BET), a two-stage framework that combines behavioral cold-start with GRPO under an investment-cost-aware reward.<n>BET reduces reasoning tokens by 55% on average while achieving overall performance improvements.
  arXiv  Detail & Related papers  (2026-05-12T06:51:31Z)
• The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits [12.316478346055618]
  Chain-of-thought reasoning is often treated as a monotone way to improve language-model accuracy by letting a model think longer.<n>We identify a countervailing effect, the coupling tax: when reasoning traces and final answers share one output-token budget, long traces can crowd out the answer they are meant to support.<n>Across GSM8K, MATH-500, and five BIG-Bench Hard tasks with Qwen3 models at three scales, non-thinking mode matches or outperforms thinking mode on GSM8K and MATH-500 at every budget up to 2048 tokens.
  arXiv  Detail & Related papers  (2026-05-08T12:54:53Z)
• Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data [57.996437077411315]
  We study the reasoning behavior of large language models (LLMs) under limited computation budgets.<n>We introduce an anytime reasoning framework and the Anytime Index, a metric that quantifies how effectively solution quality improves as reasoning tokens increase.<n> Experiments on NaturalPlan (Trip), AIME, and GPQA datasets show consistent gains across Grok-3, GPT-oss, GPT-4.1/4o, and LLaMA models.
  arXiv  Detail & Related papers  (2026-01-16T07:09:30Z)
• Steering LLM Thinking with Budget Guidance [48.65894557568655]
  Budget guidance is a method for steering the reasoning process of LLMs toward a target budget without requiring any fine-tuning.<n>Our approach introduces a lightweight predictor that models a Gamma distribution over the remaining thinking length.<n>This signal is then used to guide generation in a soft, token-level manner, ensuring that the overall reasoning trace adheres to the specified thinking budget.
  arXiv  Detail & Related papers  (2025-06-16T17:57:05Z)
• Plan and Budget: Effective and Efficient Test-Time Scaling on Large Language Model Reasoning [19.258292534503887]
  Plan-and-Budget is a model-agnostic, test-time framework that decomposes complex queries into sub-questions and allocates token budgets based on estimated complexity using adaptive scheduling.<n>Plan-and-Budget improves reasoning efficiency across a range of tasks and models, achieving up to +70% accuracy gains, tangential -39% token reduction, and +187.5% improvement in $E3$.
  arXiv  Detail & Related papers  (2025-05-22T01:56:29Z)
• Adaptive Thinking via Mode Policy Optimization for Social Language Agents [75.3092060637826]
  We propose a framework to improve the adaptive thinking ability of language agents in dynamic social interactions.<n>Our framework advances existing research in three key aspects: (1) Multi-granular thinking mode design, (2) Context-aware mode switching across social interaction, and (3) Token-efficient reasoning via depth-adaptive processing.
  arXiv  Detail & Related papers  (2025-05-04T15:39:58Z)
• DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models [30.184895117009457]
  This paper introduces Difficulty-Adaptive Slow Thinking (DAST), a novel framework that enables models to autonomously adjust the length of Chain-of-Thought (CoT) based on problem difficulty.<n>Experiments on diverse datasets and model scales demonstrate that DAST effectively mitigates overthinking while preserving reasoning accuracy on complex problems.
  arXiv  Detail & Related papers  (2025-03-06T14:23:06Z)
• Reasoning with Latent Thoughts: On the Power of Looped Transformers [52.84192961524481]
  We show that for many synthetic reasoning problems, a $k$-layer transformer looped $L$ times nearly matches the performance of a $kL$-layer non-looped model. Our empirical analysis reveals an intriguing phenomenon: looped and non-looped models exhibit scaling behavior that depends on their effective depth.
  arXiv  Detail & Related papers  (2025-02-24T18:49:05Z)
• Achieving PAC Guarantees in Mechanism Design through Multi-Armed Bandits [8.013444110633223]
  We analytically derive a class of optimal solutions to a linear program (LP) for automated mechanism design.<n>These solutions can be expressed using a set of essential variables whose cardinality is exponentially smaller than the total number of variables in the original formulation.<n>We address this by translating the evaluation of this term into a multi-armed bandit (MAB) problem.
  arXiv  Detail & Related papers  (2024-11-30T03:59:36Z)
• Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths [69.39559168050923]
  We introduce Reasoning Paths Optimization (RPO), which enables learning to reason and explore from diverse paths. Our approach encourages favorable branches at each reasoning step while penalizing unfavorable ones, enhancing the model's overall problem-solving performance. We focus on multi-step reasoning tasks, such as math word problems and science-based exam questions.
  arXiv  Detail & Related papers  (2024-10-07T06:37:25Z)
• Metareasoning in uncertain environments: a meta-BAMDP framework [1.0923877073891441]
  Finding the right $P$ can itself be framed as an optimization problem over the space of reasoning processes $P$. This paper proposes a meta Bayes-Adaptive MDP framework to handle metareasoning in environments with unknown reward/transition distributions.
  arXiv  Detail & Related papers  (2024-08-02T13:15:01Z)
• Stochastic Multi-round Submodular Optimization with Budget [7.902059578326225]
  We aim to adaptively maximize the sum, over multiple rounds, of a monotone submodular objective function defined on subsets of items. The objective function also depends on the realization of events, and the total number of items we can select over all rounds is bounded by a limited budget.
  arXiv  Detail & Related papers  (2024-04-21T18:24:43Z)
• Scalable 3D Registration via Truncated Entry-wise Absolute Residuals [65.04922801371363]
  A $3$D registration approach can process more than ten million ($107$) point pairs with over $99%$ random outliers. We call our method TEAR, as it involves minimizing an outlier-robust loss that computes Truncated Entry-wise Absolute Residuals.
  arXiv  Detail & Related papers  (2024-04-01T04:43:39Z)
• On Optimal Caching and Model Multiplexing for Large Model Inference [66.50550915522551]
  Large Language Models (LLMs) and other large foundation models have achieved noteworthy success, but their size exacerbates existing resource consumption and latency challenges. We study two approaches for mitigating these challenges: employing a cache to store previous queries and learning a model multiplexer to choose from an ensemble of models for query processing.
  arXiv  Detail & Related papers  (2023-06-03T05:01:51Z)
• Minimax Optimization with Smooth Algorithmic Adversaries [59.47122537182611]
  We propose a new algorithm for the min-player against smooth algorithms deployed by an adversary. Our algorithm is guaranteed to make monotonic progress having no limit cycles, and to find an appropriate number of gradient ascents.
  arXiv  Detail & Related papers  (2021-06-02T22:03:36Z)
• Curriculum learning for multilevel budgeted combinatorial problems [7.804994311050265]
  Multilevel optimization problems are their generalization, encompassing situations with multiple players taking decisions sequentially. We devise a value-based method to learn to solve multilevel budgeted problems involving two players in a zero-sum game over a graph. Our framework is based on a simple curriculum: if an agent knows how to estimate the value of instances with budgets up to $B$, then solving instances with budget $B+1$ can be done in time regardless of the direction of every possible afterstate.
  arXiv  Detail & Related papers  (2020-07-07T01:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.