RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

• URL: http://arxiv.org/abs/2609.03379v2
• Date: Thu, 10 Sep 2026 08:51:51 GMT
• Title: RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory
• Abstract summary: Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis.<n>RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute.
• Score: 25.81050020222202
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Repeating a small block of middle layers increases a language model's effective inference depth without adding parameters or generating extra tokens, and recent work shows that this latent recurrence improves reasoning. However, two design choices limit these gains. Each iteration sees only the previous output and cannot directly access earlier computations. Moreover, a fixed loop count wastes depth on easy inputs while leaving hard ones with too little computation. We introduce RecurTrace, which addresses both limitations using the loop's own trajectory. Specifically, Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone. A halting head then reads the loop state and predicts whether to continue, with supervision from an oracle that identifies when additional depth still reduces loss. In a controlled MathQA comparison on the same looped backbone, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute. By comparison, ACT and PonderNet collapse to one loop, and CALM reaches only 54.1% with 5.6 loops, while the stronger LoopUS-Conf and TaH-Mismatch baselines reach 55.3% at 3.2 loops and 55.7% at 2.1 loops. Finally, RecurTrace improves generation accuracy over same-budget fine-tuned baselines at 0.6B, 1.7B, 4B, and 8B, with the gain growing with model size from 0.6 to 3.4 points.

Related papers

• SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers [50.00050394179417]
  Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size.<n>We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache.<n>Through a series of ablations, we arrive at a recipe we call SMELT, which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets.
  arXiv  Detail & Related papers  (2026-09-01T14:52:57Z)
• REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation [64.84437332310516]
  We introduce textbfREER-PT, a scalable framework that extends Reverse-Engineered Reasoning (REER) to raw pre-training data.<n>REER-PT identifies continuations that are difficult to predict but can still be inferred from the preceding context, and inserts concise reasoning annotations that reconstruct the missing connection between context and continuation.<n>This sparse transformation preserves the source text and remains compatible with standard next-token prediction, avoiding online reasoning rollouts during pre-training.
  arXiv  Detail & Related papers  (2026-08-31T11:34:19Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers [0.0]
  We show a training-free rule that halts each token once its output stabilizes attains uniform depth-8 quality at 4.94 average loops.<n>A linear router trained on convergence labels harvested from the same model requires nearly full depth and yields no reduction.<n>We report average depth as a FLOP proxy with a three-point wall-clock bracket rather than a realized speedup.
  arXiv  Detail & Related papers  (2026-07-15T23:38:44Z)
• DeepLoop: Depth Scaling for Looped Transformers [91.7922038545625]
  Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds.<n>We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient.<n>The resulting method, textbfDeepLoop, keeps the Post-LN DeepNorm architecture and sets $= (2N)1/2$ and $=(8N)-1/2$ for unrolled depth.
  arXiv  Detail & Related papers  (2026-07-15T06:37:05Z)
• LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling [72.71005779366162]
  This study studies loop-count selection through a gain--cost view.<n>We instantiate this study by training LoopCoder-v2 with different loop counts, from scratch on 18T tokens, followed by matched instruction tuning and evaluation.<n> Empirically, the two-loop variant delivers broad gains over the non-looped baseline across code generation, code reasoning, agentic software engineering, and tooluse benchmarks.
  arXiv  Detail & Related papers  (2026-06-16T15:03:05Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Perturbation Dose Responses in Recursive LLM Loops: Raw Switching, Stochastic Floors, and Persistent Escape under Append, Replace, and Dialog Updates [0.0]
  We study how much injected text is needed to move a settled loop somewhere else, and whether that move lasts.<n>Under a 12,000-character tail clip, destination-coherent persistence plateaus near 16 percent and retained source-basin escape near 36 percent at dose 400.<n>A four-step falsification battery recasts the high-dose destination-coherent dip as a finite-horizon, endpoint-definition-sensitive feature.
  arXiv  Detail & Related papers  (2026-05-04T05:16:43Z)
• Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning [0.0]
  We study learned memory tokens as computational scratchpad for a single-block Universal Transformer (UT) on SudokuExtreme.<n>We find that memory tokens are empirically necessary: across all configurations tested, no configuration without memory tokens non-trivial performance.
  arXiv  Detail & Related papers  (2026-04-23T18:30:01Z)
• Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models [22.525318796588568]
  Think-at-Hard (TaH) is a dynamic latent thinking method that iterates deeper only at hard tokens.<n>TaH boosts reasoning performance across five challenging benchmarks.
  arXiv  Detail & Related papers  (2025-11-11T18:57:02Z)
• DeepPrune: Parallel Scaling without Inter-trace Redundancy [53.62015294143274]
  Over 80% of parallel reasoning traces yield identical final answers, representing substantial wasted computation.<n>We propose DeepPrune, a novel framework that enables efficient parallel scaling through dynamic pruning.<n>Our work establishes a new standard for efficient parallel reasoning, making high-performance reasoning more efficient.
  arXiv  Detail & Related papers  (2025-10-09T17:24:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.