LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

• URL: http://arxiv.org/abs/2609.17184v1
• Date: Tue, 15 Sep 2026 13:43:31 GMT
• Title: LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
• Abstract summary: Looped Transformers incur higher decoding latency than standard Transformer models because shared weights are accessed at every recurrent depth.<n>We propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers.<n>Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83$times$ inference speedup across diverse Looped Transformers.
• Score: 16.277740642154402
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer models of comparable parameter size because shared weights are accessed at every recurrent depth. To improve decoding efficiency, self-speculative decoding is particularly well suited to Looped Transformers, as their intermediate recurrent states can directly provide draft predictions without an auxiliary draft model. We therefore propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers. LoopSpec extracts draft tokens from early recurrent states and operates in a pipelined manner, overlapping draft generation of future tokens with target verification of the current token. To improve draft accuracy without excessive compute overhead, we introduce a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes. Furthermore, we derive the optimal proposal depths in closed form and show the prediction matches measurement. Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83$\times$ inference speedup across diverse Looped Transformers.

Related papers

• SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers [50.00050394179417]
  Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size.<n>We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache.<n>Through a series of ablations, we arrive at a recipe we call SMELT, which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets.
  arXiv  Detail & Related papers  (2026-09-01T14:52:57Z)
• Full-bandwidth transformer [19.176565344686043]
  Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.<n>We introduce the emphfull-bandwidth transformer, which widens this channel with emphlatent feedback<n>Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget.
  arXiv  Detail & Related papers  (2026-08-09T19:59:45Z)
• LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization [14.897282681862324]
  Speculative decoding accelerates large language model inference by drafting multiple tokens for parallel verification.<n>We develop textitLibraSpec, a training-free and plug-and-play algorithm that iteratively determines the speculative length using drafter confidence scores.
  arXiv  Detail & Related papers  (2026-08-09T14:15:14Z)
• N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation [68.47358899451255]
  N-vium is a mixture-of-exits transformer that partially parallelizes computation across depth on standard hardware.<n>N-vium attaches prediction heads at multiple depths and defines the next-token distribution as a learned mixture over these exits.
  arXiv  Detail & Related papers  (2026-05-13T08:46:17Z)
• DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streams [63.27233749591346]
  Transformer-based models have dramatically increased their size and parameter count to tackle increasingly complex tasks.<n>Stream data inference is typically performed over a sliding temporal window, leading to highly redundant computations.<n>We propose the Deep Continual Transformer (DeepCoT), a redundancy-free encoder-only model that can be applied over existing deep encoder architectures with minimal changes.
  arXiv  Detail & Related papers  (2025-11-21T16:15:43Z)
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding [66.40658898418316]
  We present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion language model (DLM) to produce multi-token drafts in a single forward pass.<n>Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.
  arXiv  Detail & Related papers  (2025-09-28T07:00:15Z)
• Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers [24.109312575970456]
  We propose a simple framework to enable the offthe-shelf pre-trained transformers to process much longer sequences. Our method divides each long-sequence input into a batch of chunks, then aligns the interchunk information during the encoding steps. We learn an effective hidden selection policy, which regards the decoders of transformers as environments.
  arXiv  Detail & Related papers  (2023-08-25T05:52:05Z)
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing [112.2208052057002]
  We propose Funnel-Transformer which gradually compresses the sequence of hidden states to a shorter one. With comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on a wide variety of sequence-level prediction tasks.
  arXiv  Detail & Related papers  (2020-06-05T05:16:23Z)
• On Sparsifying Encoder Outputs in Sequence-to-Sequence Models [90.58793284654692]
  We take Transformer as the testbed and introduce a layer of gates in-between the encoder and the decoder. The gates are regularized using the expected value of the sparsity-inducing L0penalty. We investigate the effects of this sparsification on two machine translation and two summarization tasks.
  arXiv  Detail & Related papers  (2020-04-24T16:57:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.