Prefix Sliding for efficient test-time scaling

• URL: http://arxiv.org/abs/2608.26070v1
• Date: Wed, 26 Aug 2026 17:37:15 GMT
• Title: Prefix Sliding for efficient test-time scaling
• Abstract summary: Test-time scaling uses extra test-time compute to improve performance.<n>We find most intermediate reasoning tokens lose importance as the model continues reasoning.<n>We propose Prefix Sliding, which discards tokens during reasoning that are not part of the prefix or the window of the last few thousand tokens.
• Score: 148.8911065058742
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time scaling uses extra test-time compute to improve performance, such as letting language models reason longer when solving a problem. As models keep the entire reasoning trace in memory via full attention, hard tasks that need long thinking can be prohibitively expensive. However, we find most intermediate reasoning tokens lose importance as the model continues reasoning. This calls into question whether retaining them is worth the cost. Based on this insight, we propose Prefix Sliding, which discards tokens during reasoning that are not part of the prefix or the window of the last few thousand tokens. The prefix has key instructions and tools available to the model, while the most recent tokens are the current reasoning the model is working on. This caps the total memory requirement regardless of how long the model reasons, allowing for efficient long-horizon test-time scaling. Without training, Prefix Sliding can make existing models 3x faster while maintaining performance. Training with Prefix Sliding using reinforcement learning can achieve better performance by enabling scaling to reasoning traces beyond a hundred thousand tokens. Ablations show Prefix Sliding outperforms summarizing intermediate tokens or vanilla sliding window. Our code is at https://github.com/Muennighoff/prefix-sliding

Related papers

• Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference [58.46323651532913]
  We study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights.<n>During sleep, the model performs $N$ offline recurrent passes over the accumulated context and updates the fast weights in its state-space model.<n>We show that increasing sleep duration $N$ for our models improves performance, with the largest gains on examples that require deeper reasoning.
  arXiv  Detail & Related papers  (2026-05-25T17:55:39Z)
• Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production [55.76222360698305]
  We explore a class of supervised training objectives that allow a language model to dynamically and autonomously scale the number of compute steps used for each input token.<n>For any token, the model can request additional compute steps by emitting a don't know> output.<n>We find that the CYB model requests additional steps when doing so improves accuracy, and the model adapts its processing time to token-level complexity and context.
  arXiv  Detail & Related papers  (2025-10-13T21:07:05Z)
• Scaling Speculative Decoding with Lookahead Reasoning [11.349400331288257]
  Token-level speculative decoding (SD) helps, but its benefit is capped.<n>We develop Lookahead Reasoning, which exploits a second, step-level layer of parallelism.<n>Lookahead Reasoning improves the speedup of SD from 1.4x to 2.1x while preserving answer quality.
  arXiv  Detail & Related papers  (2025-06-24T17:48:10Z)
• Learning a Continue-Thinking Token for Enhanced Test-Time Scaling [15.231981094907386]
  Test-time scaling has emerged as an effective approach for improving language model performance by utilizing additional compute at inference time.<n>Recent studies have shown that overriding end-of-thinking tokens can extend reasoning steps and improve accuracy.<n>In this work, we explore whether a dedicated continue-thinking token can be learned to trigger extended reasoning.
  arXiv  Detail & Related papers  (2025-06-12T20:28:54Z)
• HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving [2.7433801927536074]
  Early-Exit Large Language Models (EE-LLMs) enable high throughput inference by allowing tokens to exit early at intermediate layers.<n>Existing EE-LLM frameworks rely on a single model and therefore, their token generation latencies are bottlenecked.<n>We propose $textitHELIOS, a framework that improves both token generation latency and batch sizes.
  arXiv  Detail & Related papers  (2025-04-14T21:30:43Z)
• Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach [70.44265766483633]
  We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space.<n>Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time.<n>We show that the resulting model can improve its performance on reasoning benchmarks, sometimes dramatically.
  arXiv  Detail & Related papers  (2025-02-07T18:55:02Z)
• Think before you speak: Training Language Models With Pause Tokens [73.61375226378712]
  Language models generate responses by producing a series of tokens in immediate succession. What if instead we were to let the model manipulate say, $K+10$ hidden vectors, before it outputs the $(K+1)th$ token? We operationalize this idea by performing training and inference on language models with a (learnable) $textitpause$ token.
  arXiv  Detail & Related papers  (2023-10-03T17:32:41Z)
• Confident Adaptive Language Modeling [95.45272377648773]
  CALM is a framework for dynamically allocating different amounts of compute per input and generation timestep. We demonstrate the efficacy of our framework in reducing compute -- potential speedup of up to $times 3$ -- while provably maintaining high performance.
  arXiv  Detail & Related papers  (2022-07-14T17:00:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.