T^2MLR: Transformer with Temporal Middle-Layer Recurrence

• URL: http://arxiv.org/abs/2607.15178v2
• Date: Fri, 17 Jul 2026 05:00:38 GMT
• Title: T^2MLR: Transformer with Temporal Middle-Layer Recurrence
• Abstract summary: We develop a transformers-based latent reasoning architecture that fuses a cached middle layer representation from the previous token directly into an earlier layer of the current token position.<n>Applying recurrence to only a localized middle-layer block (as little as 20% of the network) often outperforms full-layer recurrence.<n>These results suggest that effective latent reasoning in Transformers does not require looping over all layers as in previous works, but can instead emerge more strongly from targeted middle-layer recurrence.
• Score: 28.818755729513793
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer reasoning is limited by autoregressive decoding, which repeat edly compresses rich hidden computation through token space and makes it difficult for intermediate reasoning states to persist across time. We in troduce Transformers with Temporal Middle-Layer Recurrence (T2MLR), a transformers-based latent reasoning architecture that fuses a cached middle layer representation from the previous token directly into an earlier layer of the current token position, enabling abstract intermediate computation to persist across decoding steps with little inference overhead. Across natural-language pretraining and multi-hop reasoning finetuning, T2MLR consistently outperforms data- and parameter-matched Transformer base lines. Moreover, applying recurrence to only a localized middle-layer block (as little as 20% of the network) often outperforms full-layer recurrence. Im portantly, T2MLR does not require pretraining from scratch: retrofitting the recurrent pathway into an existing pretrained 1.7B Transformer and briefly finetuning substantially improves math reasoning, lowering the barrier to practical adoption. These results suggest that effective latent reasoning in Transformers does not require looping over all layers as in previous works, but can instead emerge more strongly from targeted middle-layer recurrence.

Related papers

• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models [75.80275843320511]
  We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.<n>We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining.
  arXiv  Detail & Related papers  (2026-07-09T07:37:59Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Transformers with Selective Access to Early Representations [6.554898376530473]
  We introduce Selective Access Transformer (SATFormer), which preserves the first-layer value pathway while controlling access with a context-dependent gate.<n>SATFormer consistently improves validation loss and zero-shot accuracy over the static value-residual and Transformer baselines.<n> Gate analyses suggest sparse, depth-dependent, head-specific, and category-sensitive access patterns.
  arXiv  Detail & Related papers  (2026-05-05T16:38:29Z)
• The Recurrent Transformer: Greater Effective Depth and Efficient Decoding [48.9323408950142]
  We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations.<n>We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-gressive recurrent updates under mild assumptions.
  arXiv  Detail & Related papers  (2026-04-23T02:12:58Z)
• DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streams [63.27233749591346]
  Transformer-based models have dramatically increased their size and parameter count to tackle increasingly complex tasks.<n>Stream data inference is typically performed over a sliding temporal window, leading to highly redundant computations.<n>We propose the Deep Continual Transformer (DeepCoT), a redundancy-free encoder-only model that can be applied over existing deep encoder architectures with minimal changes.
  arXiv  Detail & Related papers  (2025-11-21T16:15:43Z)
• TNT: Improving Chunkwise Training for Test-Time Memorization [62.78875147721906]
  Recurrent neural networks (RNNs) with deep test-time memorization modules, such as Titans and TTT, represent a promising, linearly-scaling paradigm distinct from Transformers.<n>We introduce TNT, a novel training paradigm that decouples training efficiency from inference performance through a two-stage process.<n>TNT achieves a substantial acceleration in training speed-up to 17 times faster than the most accurate baseline configuration.
  arXiv  Detail & Related papers  (2025-11-10T17:45:09Z)
• Intra-Layer Recurrence in Transformers for Language Modeling [0.03320194947871346]
  Intra-Layer Recurrence (ILR) is a more targeted approach that applies recurrence selectively to individual layers within a single forward pass.<n>Our experiments show that allocating more iterations to earlier layers yields optimal results.
  arXiv  Detail & Related papers  (2025-05-03T16:16:55Z)
• Finetuning Pretrained Transformers into RNNs [81.72974646901136]
  Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. A linear-complexity recurrent variant has proven well suited for autoregressive generation. This work aims to convert a pretrained transformer into its efficient recurrent counterpart.
  arXiv  Detail & Related papers  (2021-03-24T10:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.