Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions

• URL: http://arxiv.org/abs/2608.07968v2
• Date: Tue, 11 Aug 2026 02:23:08 GMT
• Title: Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions
• Abstract summary: We introduce an exam-style evaluation framework for studying this setting.<n>We find that models fail to allocate a shared budget strategically across questions of varying difficulties and values.<n>These findings establish global budget allocation as a distinct capability that is not captured by conventional per-question evaluation.
• Score: 59.21329728895449
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning language models increasingly use test-time compute to improve performance, but existing evaluations typically study this compute one question at a time. Yet when multiple problems share an end-to-end cost or latency constraint, models must decide how to divide limited inference compute among them. We introduce an exam-style evaluation framework for studying this setting, in which a model must distribute one shared token budget across questions with different difficulty and point values to maximize its total score. Across several open and frontier reasoning models, we find that models fail to allocate a shared budget strategically across questions of varying difficulties and values. Models behave largely as greedy sequential solvers: they prioritize questions by presentation order, front-load effort on early questions, and remain insensitive to value, with these tendencies becoming more pronounced as the number of questions grows. Explicit planning prompts spread compute more evenly but do not produce value- or difficulty-aware prioritization. The same behavioral pattern extends from mathematical to code reasoning. These findings establish global budget allocation as a distinct capability that is not captured by conventional per-question evaluation and remains a challenge for current reasoning models.

Related papers

• How Inference Compute Shapes Frontier LLM Evaluation [2.7633794124150426]
  We evaluate up to 12 frontier language models on seven challenging benchmarks spanning software engineering, mathematics, medicine, and cybersecurity.<n>We find that newer models reach higher performance at large budgets, where they unlock harder tasks and solve them more reliably.<n>We argue that evaluations should report capability as a function of inference-time compute, specify protocol choices explicitly, and compare model generations over a large shared compute range.
  arXiv  Detail & Related papers  (2026-06-16T13:40:53Z)
• TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints [0.0]
  We introduce TRIAGE, a framework for testing language models for metacognitive control.<n>We find that current language models exhibit substantial gaps in prospective metacognitive control.
  arXiv  Detail & Related papers  (2026-05-13T12:10:05Z)
• TopBench: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering [80.93487993878836]
  A common class of real-world queries is implicitly predictive, requiring the inference of unobserved answers from historical patterns rather than mere retrieval.<n>These queries introduce two challenges: recognizing latent intent and reliable predictive reasoning over massive tables.<n>We introduce TopBench, a benchmark consisting of 779 samples across four sub-tasks, ranging from single-point prediction to decision making, treatment effect analysis, and complex reasoning.
  arXiv  Detail & Related papers  (2026-04-30T16:22:51Z)
• MorphoBench: A Benchmark with Difficulty Adaptive to Model Reasoning [61.04601861108966]
  We propose MorphoBench, a benchmark that incorporates multidisciplinary questions to evaluate the reasoning capabilities of large models.<n>MorphoBench adaptively modifies the analytical challenge of questions by leveraging key statements generated during the model's reasoning process.<n>We have gathered over 1,300 test questions and iteratively adjusted the difficulty of MorphoBench based on the reasoning capabilities of models such as o3 and GPT-5.
  arXiv  Detail & Related papers  (2025-10-16T03:30:56Z)
• T$^2$: An Adaptive Test-Time Scaling Strategy for Contextual Question Answering [49.5489716597489]
  We present T$2$: Think-to-Think, a novel framework that dynamically adapts reasoning depth based on question complexity.<n>T$2$ works through four key steps: decomposing questions into structural elements, generating similar examples with candidate reasoning strategies, evaluating these strategies against multiple criteria, and applying the most appropriate strategy to the original question.
  arXiv  Detail & Related papers  (2025-05-23T03:18:02Z)
• THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Models [65.39456695678713]
  We introduce approximate measures of problem-level difficulty and demonstrate that a clear relationship between problem difficulty and optimal token spend exists.<n>We find that in general, reasoning models are poorly calibrated, particularly on easy problems.<n>We introduce THOUGHTTERMINATOR, a training-free black box decoding technique that significantly improves reasoning model calibration.
  arXiv  Detail & Related papers  (2025-04-17T22:16:30Z)
• Reliable and Efficient Amortized Model-based Evaluation [57.6469531082784]
  The average score across a wide range of benchmarks provides a signal that helps guide the use of language models in practice.<n>A popular attempt to lower the cost is to compute the average score on a subset of the benchmark.<n>This approach often renders an unreliable measure of LM performance because the average score is often confounded with the difficulty of the questions in the benchmark subset.<n>We train a model that predicts question difficulty from its content, enabling a reliable measurement at a fraction of the cost.
  arXiv  Detail & Related papers  (2025-03-17T16:15:02Z)
• Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation [12.638577140117702]
  We show that uncertainty features contribute substantially to difficulty prediction, where difficulty is inversely proportional to the number of students who can correctly answer a question.<n>In addition to showing the value of our approach, we also observe that our model achieves state-of-the-art results on the USMLE and CMCQRD publicly available datasets.
  arXiv  Detail & Related papers  (2024-12-16T14:55:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.