LoopMTP: A looped transformer guided by latent multi-token prediction

• URL: http://arxiv.org/abs/2608.03624v1
• Date: Tue, 04 Aug 2026 13:13:09 GMT
• Title: LoopMTP: A looped transformer guided by latent multi-token prediction
• Abstract summary: Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing.<n>textscLoopMTP improves average accuracy by up to 8.1% (relative) over the non-looped baseline.
• Score: 2.2694459418593698
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning. By reusing one stack of layers across $T$ iterations, they attain the effective depth and reasoning capabilities of larger models at a fixed parameter count. Yet existing approaches suffer from latent overthinking and undifferentiated computation, largely because intermediate representations receive no guidance across loops. Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing. We propose \textsc{LoopMTP}, which links the two through a structural correspondence in latent space: a model that loops $T$ times can anticipate $T$ future tokens. \textsc{LoopMTP} realizes this by softly aligning the hidden state of loop $t$ with the embedding of the token $t$ steps ahead, while a lightweight gate preserves useful information across iterations. \textsc{LoopMTP} improves average accuracy by up to 8.1\% (relative) over the non-looped baseline, with training remaining stable for up to 15 loops.

Related papers

• T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning With Dynamic Routing [31.346695540393657]
  Looped Transformers have recently demonstrated strong performance in both reasoning and language tasks.<n>In this work, we propose textbfdynamic token-choice routing for looped transformers.<n>We use a dynamic router to decide whether a token should continue recursing or exit early, allowing simple tokens to bypass unnecessary computation.
  arXiv  Detail & Related papers  (2026-09-14T07:41:37Z)
• HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation [56.14416807766864]
  Homomorphic encryption allows a server to run a language model directly on encrypted user prompts.<n>Existing approaches fix the iteration counts uniformly across the model rather than tailoring them to each site's error tolerance.<n>We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonline iteration counts learnable.
  arXiv  Detail & Related papers  (2026-09-01T18:02:02Z)
• SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers [50.00050394179417]
  Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size.<n>We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache.<n>Through a series of ablations, we arrive at a recipe we call SMELT, which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets.
  arXiv  Detail & Related papers  (2026-09-01T14:52:57Z)
• Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching [33.509836193149795]
  A main promise of looped language models (LMs) is depth-adaptive computation.<n>We introduce continuous depth (CDB) which schedules at granularity of individual loop queues.<n>On Ouro 1.4B and Huginn 3.5B, CDB can realize $99%$ of the theoretical maximum speed-up.
  arXiv  Detail & Related papers  (2026-08-10T11:20:14Z)
• DeepLoop: Depth Scaling for Looped Transformers [91.7922038545625]
  Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds.<n>We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient.<n>The resulting method, textbfDeepLoop, keeps the Post-LN DeepNorm architecture and sets $= (2N)1/2$ and $=(8N)-1/2$ for unrolled depth.
  arXiv  Detail & Related papers  (2026-07-15T06:37:05Z)
• Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers [24.474589810964158]
  Language models typically reason via explicit chain-of-thought (CoT)<n>Latent CoT offers an alternative: it performs multi-step reasoning in the model's hidden states.<n>Looped Transformers are a natural fit for latent reasoning.
  arXiv  Detail & Related papers  (2026-06-30T14:58:53Z)
• On the Residual Scaling of Looped Transformers: Stability and Transferability [31.27468588849646]
  We show that $1/N$ scaling improves trainability and yields better loss than $1/!sqrtN$ scaling across loop counts.<n>Experiments on looped Transformers confirm that $1/N$ scaling improves trainability and yields better loss than $1/!sqrtN$ scaling across loop counts.
  arXiv  Detail & Related papers  (2026-06-16T22:39:13Z)
• LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling [72.71005779366162]
  This study studies loop-count selection through a gain--cost view.<n>We instantiate this study by training LoopCoder-v2 with different loop counts, from scratch on 18T tokens, followed by matched instruction tuning and evaluation.<n> Empirically, the two-loop variant delivers broad gains over the non-looped baseline across code generation, code reasoning, agentic software engineering, and tooluse benchmarks.
  arXiv  Detail & Related papers  (2026-06-16T15:03:05Z)
• Incremental BPE Tokenization [12.857819598120676]
  We propose a novel algorithm for incremental Byte Pair.<n>Model (BPE) tokenization.<n>We demonstrate significant latency reductions over OpenAI's tiktoken on pathological inputs.
  arXiv  Detail & Related papers  (2026-05-29T04:04:32Z)
• Transformers Provably Learn to Internalize Chain-of-Thought [65.41010769606844]
  Implicit Chain-of-Thought (ICoT) trains models to internalize intermediate steps within their hidden states.<n>Log-ICoT removes thinking tokens one at a time, reducing the number of stages from linear in $k$ to logarithmic.<n>Experiments on multi-layer transformers confirm the theory and visualize how reasoning is progressively absorbed into deeper layers.
  arXiv  Detail & Related papers  (2026-05-27T15:17:06Z)
• LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation [9.943277041891788]
  We introduce LoopFormer, a looped Transformer trained on variable-length trajectories to enable budget-conditioned reasoning.<n>Our core contribution is a shortcut-consistency training scheme that aligns trajectories of different lengths.<n>LoopFormer demonstrates robust performance on language modeling and reasoning benchmarks even under aggressive compute constraints.
  arXiv  Detail & Related papers  (2026-02-11T23:58:28Z)
• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems [57.58801785642868]
  Chain of thought (CoT) is a highly effective method to improve the accuracy of large language models (LLMs) on arithmetics and symbolic reasoning tasks. This work provides a theoretical understanding of the power of CoT for decoder-only transformers through the lens of expressiveness.
  arXiv  Detail & Related papers  (2024-02-20T10:11:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.