Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior

• URL: http://arxiv.org/abs/2605.26797v1
• Date: Tue, 26 May 2026 10:10:26 GMT
• Title: Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
• Abstract summary: Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
• Score: 107.2098567818173
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token. Because this source state is already computed during ordinary decoding, LRT adds a cross-layer recurrent latent pathway across positions without inserting pause tokens or extra depth loops, and the standard attention mechanism and KV-cache interface are preserved. To pretrain this recurrence at scale without sequentially unrolling the transformer, we introduce interleaved parallel training: a single full-sequence initialization forward pass builds a shared buffer; then disjoint position subsets are refined in parallel and written back, so that all tokens receive recurrent-memory-aware supervision at roughly 2 times baseline compute. Across nanochat style backbones and a wide range of tokens-per-parameter budgets, LRT improves both language-modeling loss and in-context learning under matched effective compute while adding as little as 0.3% parameters.

Related papers

• Full-bandwidth transformer [19.176565344686043]
  Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.<n>We introduce the emphfull-bandwidth transformer, which widens this channel with emphlatent feedback<n>Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget.
  arXiv  Detail & Related papers  (2026-08-09T19:59:45Z)
• T^2MLR: Transformer with Temporal Middle-Layer Recurrence [28.818755729513793]
  We develop a transformers-based latent reasoning architecture that fuses a cached middle layer representation from the previous token directly into an earlier layer of the current token position.<n>Applying recurrence to only a localized middle-layer block (as little as 20% of the network) often outperforms full-layer recurrence.<n>These results suggest that effective latent reasoning in Transformers does not require looping over all layers as in previous works, but can instead emerge more strongly from targeted middle-layer recurrence.
  arXiv  Detail & Related papers  (2026-07-16T16:33:59Z)
• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models [75.80275843320511]
  We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.<n>We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining.
  arXiv  Detail & Related papers  (2026-07-09T07:37:59Z)
• Déjà View: Looping Transformers for Multi-View 3D Reconstruction [109.98721386523641]
  Recent feed-forward 3D reconstruction transformers have scaled to over a billion parameters.<n>We show that model depth partially buys iteration, paid for inefficiently in unique parameters, and instead make that iteration explicit in architecture.<n>Our model, DéjView, applies a single looped transformer block recurrently to per-view features for K refinement steps.
  arXiv  Detail & Related papers  (2026-05-28T16:47:48Z)
• Replacement Learning: Training Neural Networks with Fewer Parameters [15.09968642484538]
  Replacement Learning (RepL) is a training-time paradigm that reduces full-depth redundancy by replacing selected blocks.<n>RepL preserves local contextual continuity while avoiding unnecessary full-layer.<n>We show that RepL reduces trainable parameters, GPU memory usage, and training time while matching or surpassing standard end-to-end training.
  arXiv  Detail & Related papers  (2026-05-19T08:34:31Z)
• Parallel Recursive LSTM [26.883902799977964]
  We introduce the Parallel Recursive LSTM, a hierarchical recurrent architecture that replaces left-to-right recurrence with nonlinear state composition over a balanced computation tree.<n>PR-LSTM achieves strong sequence-length generalization on formal-language benchmarks, solving more tasks than standard RNN, LSTM, and Transformer baselines, while avoiding the quadratic scaling of attention.
  arXiv  Detail & Related papers  (2026-05-16T18:28:59Z)
• N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation [68.47358899451255]
  N-vium is a mixture-of-exits transformer that partially parallelizes computation across depth on standard hardware.<n>N-vium attaches prediction heads at multiple depths and defines the next-token distribution as a learned mixture over these exits.
  arXiv  Detail & Related papers  (2026-05-13T08:46:17Z)
• The Recurrent Transformer: Greater Effective Depth and Efficient Decoding [48.9323408950142]
  We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations.<n>We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-gressive recurrent updates under mild assumptions.
  arXiv  Detail & Related papers  (2026-04-23T02:12:58Z)
• Routing without Forgetting [20.60324059904291]
  We recast continual learning in transformers as a routing problem.<n>Instead of storing or merging task-specific prompts, RwF generates dynamic prompts through single-step associative retrieval.<n>On Split-ImageNet-R and Split-ImageNet-S, RwF outperforms prior prompt-based approaches by a large margin, even in few-shot learning regimes.
  arXiv  Detail & Related papers  (2026-03-10T12:23:46Z)
• Rethinking Weight Tying: Pseudo-Inverse Tying for Stable LM Training and Updates [22.84428628659889]
  Pseudo-Inverse Tying guarantees a pseudo-inverse-consistent interface throughout training.<n>We evaluate PIT on on-device models spanning 256M-1.3B parameters across pretraining and adaptation.
  arXiv  Detail & Related papers  (2026-02-04T13:44:53Z)
• GSPN-2: Efficient Parallel Sequence Modeling [101.33780567131716]
  Generalized Spatial Propagation Network (GSPN) addresses this by replacing quadratic self-attention with a line-scan propagation scheme.<n>GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications.
  arXiv  Detail & Related papers  (2025-11-28T07:26:45Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation [79.1669476932147]
  Vision-and-Language Navigation (VLN) is a task that an agent is required to follow a language instruction to navigate to the goal position. Recent Transformer-based VLN methods have made great progress benefiting from the direct connections between visual observations and the language instruction. We introduce Multimodal Transformer with Variable-length Memory (MTVM) for visually-grounded natural language navigation.
  arXiv  Detail & Related papers  (2021-11-10T16:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.