Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

• URL: http://arxiv.org/abs/2608.04001v1
• Date: Tue, 04 Aug 2026 17:57:20 GMT
• Title: Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
• Abstract summary: We develop a systematic account of test-time scaling along three axes.<n>First, we formalize test-time scaling as budgeted inference over the implicit prefix tree of an autoregressive model.<n>Second, we treat the evaluated object as the entire inference system.
• Score: 7.838210213616623
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models can solve substantially harder reasoning problems with more inference-time compute. The term "test-time scaling," however, now covers diverse inference algorithms that extend deliberation along a single trajectory, sample completed candidates and aggregate them through voting or verification, or search over unfinished partial states. These algorithms differ in their statistical structure, compute accounting, and failure modes. Treating these procedures as interchangeable under a single scalar "budget," or reporting accuracy without the inference protocol that produced it, makes results difficult to compare across studies. We develop a systematic account of test-time scaling along three axes. First, we formalize test-time scaling as budgeted inference over the implicit prefix tree of an autoregressive model and distinguish three structural regimes: single-trajectory sequential scaling, leaf-level scaling with terminal reduction, and prefix-level scaling. Second, we treat the evaluated object as the entire inference system and develop evaluation principles that separate end-to-end system performance from candidate-bank diagnostics. We introduce an evaluation profile whose coordinates and simple functionals recover or bound common repeated-sampling metrics, and prescribe protocol-matched reporting of compute and uncertainty. Third, we specify reproducibility requirements for inference protocols, distinguishing exact replay from distributional reproducibility and identifying the artifacts needed to support each. We also organize the open-weight reasoning ecosystem by model-side and interface mechanisms, apply these principles to broad-knowledge, symbolic-reasoning, and competition-mathematics benchmarks, and assemble over 2 billion full reasoning traces for release with progressively richer verifier and token-level signals.

Related papers

• When LLM Meets Tree Search: A Systematic View of Inference as Search in Large Language Models [92.71325249013535]
  Test-Time Scaling (TTS) has emerged as an important direction for improving reasoning by allocating inference-time compute to a fixed model prior.<n>This survey systematizes recent progress in tree-search-based reasoning, viewing inference as instance-specific optimization rather than decoding.
  arXiv  Detail & Related papers  (2026-08-31T07:49:56Z)
• MedSegBenchmarker: A Raw-Count-First Framework for Controlled 2D Medical Image Segmentation Benchmarks [1.0230975056328007]
  We present MEDSEGBENCHMARKER (MSB), a configuration-driven framework for controlled benchmarking of 2D datasets.<n>MSB exports sample- and class-level pixel counts and predictions together with the evaluation context.<n>We demonstrate MSB in a case study involving three heterogeneous 2D datasets and multiple vision models evaluated at 256- and 512-pixel input resolutions.
  arXiv  Detail & Related papers  (2026-08-30T09:20:38Z)
• The RAT: A Unified Bayesian Model for RAG Evaluation [8.650407684623714]
  We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness.<n>We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators.<n>We extend the model to incorporate LLM-as-a-judge annotations as calibrated noisy observations.
  arXiv  Detail & Related papers  (2026-08-25T15:54:30Z)
• Intent-Level Quantum Programming with Assertion-Guided Execution and Inspectable Intermediate Representation [0.22183164865527472]
  We present a quantum domain-specific language (Q) that addresses problems through three mechanisms.<n>First-class IR introspection, exposing circuit width, wire mapping, operation order, and measurement intent for developer inspection prior to execution.<n> assertion-guided modality inference, where the engine examines declared verification properties to automatically select sampling, statevector, or dual execution.
  arXiv  Detail & Related papers  (2026-07-31T18:21:32Z)
• Forethought: Verifiable Reasoning from Neurosymbolic Primitive Programming [0.764671395172401]
  Forethought is a neurosymbolic reasoning system that treats reasoning as an explicit, verifiable program.<n>It improves base-model accuracy by about 30% relative and outperforms vanilla prompting, reinforcement learning scaffolds, and prompt-evolution methods.
  arXiv  Detail & Related papers  (2026-07-05T03:20:23Z)
• Scaling Test-Time Compute for Agentic Coding [126.72747643609274]
  We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories.<n>Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes.<n>Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0.
  arXiv  Detail & Related papers  (2026-04-16T17:39:33Z)
• CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation [18.187891198793455]
  CausalReasoningBenchmark is a benchmark of 173 queries across 138 real-world datasets.<n>It distinguishes failures in causal reasoning from errors in numerical execution.
  arXiv  Detail & Related papers  (2026-02-24T05:44:25Z)
• A Single Revision Step Improves Token-Efficient LLM Reasoning [3.344806691289323]
  We introduce Packet-Conditioned Revision (PACER), a training-free, inference-only framework for large language models.<n>PACER enables reasoning traces to revise their conclusions through a structured coordination step.<n>On challenging competitive math benchmarks, PACER matches or exceeds the accuracy of 256-sample majority voting.
  arXiv  Detail & Related papers  (2026-02-02T21:28:42Z)
• Accelerate Speculative Decoding with Sparse Computation in Verification [49.74839681322316]
  Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel.<n>Existing sparsification methods are designed primarily for standard token-by-token autoregressive decoding.<n>We propose a sparse verification framework that jointly sparsifies attention, FFN, and MoE components during the verification stage to reduce the dominant computation cost.
  arXiv  Detail & Related papers  (2025-12-26T07:53:41Z)
• Adaptive Test-Time Reasoning via Reward-Guided Dual-Phase Search [62.1546099504045]
  We propose a dual-phase test-time scaling framework that separates reasoning into planning and execution.<n>Specifically, we decompose reasoning trajectories and develop reward models for each phase, enabling the search to explore and prune plans and executions separately.<n> Experiments on both mathematical reasoning and code generation benchmarks demonstrate that our approach consistently improves accuracy while reducing computation redundant.
  arXiv  Detail & Related papers  (2025-09-29T19:27:23Z)
• seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs [1.0519693622157462]
  We introduce seqBench, a benchmark for probing sequential reasoning limits in Large Language Models (LLMs)<n>We find that even top-performing models systematically fail on seqBench's structured reasoning tasks despite minimal search complexity.
  arXiv  Detail & Related papers  (2025-09-21T01:32:13Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Variable Importance Matching for Causal Inference [73.25504313552516]
  We describe a general framework called Model-to-Match that achieves these goals. Model-to-Match uses variable importance measurements to construct a distance metric. We operationalize the Model-to-Match framework with LASSO.
  arXiv  Detail & Related papers  (2023-02-23T00:43:03Z)
• Tractable Inference in Credal Sentential Decision Diagrams [116.6516175350871]
  Probabilistic sentential decision diagrams are logic circuits where the inputs of disjunctive gates are annotated by probability values. We develop the credal sentential decision diagrams, a generalisation of their probabilistic counterpart that allows for replacing the local probabilities with credal sets of mass functions. For a first empirical validation, we consider a simple application based on noisy seven-segment display images.
  arXiv  Detail & Related papers  (2020-08-19T16:04:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.