Full-bandwidth transformer

• URL: http://arxiv.org/abs/2608.08888v1
• Date: Sun, 09 Aug 2026 19:59:45 GMT
• Title: Full-bandwidth transformer
• Abstract summary: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.<n>We introduce the emphfull-bandwidth transformer, which widens this channel with emphlatent feedback<n>Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget.
• Score: 19.176565344686043
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding steps remains narrow: only the sampled token returns to the bottom of the stack, while the top-layer hidden state is discarded. We introduce the \emph{full-bandwidth transformer}, which widens this channel with \emph{latent feedback}: at each decoding step, the previous top-layer hidden state is fused with the sampled token embedding through a gated linear unit and fed back as the next input. Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget, while preserving the standard transformer architecture, KV cache, and language-modeling objective. To train full-bandwidth transformers without losing parallel teacher forcing, we use a scheduled multi-pass objective that introduces latent feedback late in pretraining and mixes a small fraction of deeper feedback passes for stability. We train 1B-parameter full-bandwidth transformers up to 400B tokens and find that latent feedback improves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance. With negligible per-token decoding overhead, full-bandwidth transformers match or approach standard transformers trained with roughly $1.5\times$ more tokens, and manage to produce shorter reasoning traces at equal or better accuracy.

Related papers

• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models [75.80275843320511]
  We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.<n>We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining.
  arXiv  Detail & Related papers  (2026-07-09T07:37:59Z)
• The Context-Ready Transformer [0.0]
  We introduce the context-ready transformer, a new recurrent neural network architecture built from a D-layer transformer block.<n>A correction network combines the previous position's block output with the current token embedding, so the tokenenters the block already contextualized.<n>A pretrained transformer can also be converted to a context-ready model by adding a zero-d correction.
  arXiv  Detail & Related papers  (2026-06-25T20:39:26Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation [68.47358899451255]
  N-vium is a mixture-of-exits transformer that partially parallelizes computation across depth on standard hardware.<n>N-vium attaches prediction heads at multiple depths and defines the next-token distribution as a learned mixture over these exits.
  arXiv  Detail & Related papers  (2026-05-13T08:46:17Z)
• The Recurrent Transformer: Greater Effective Depth and Efficient Decoding [48.9323408950142]
  We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations.<n>We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-gressive recurrent updates under mild assumptions.
  arXiv  Detail & Related papers  (2026-04-23T02:12:58Z)
• Block Transformer: Global-to-Local Language Modeling for Fast Inference [41.361135177431336]
  Self-attention requires the key-value cache of all previous sequences to be retrieved from memory at every decoding step to retrieve context information. We introduce the Block Transformer which adopts hierarchical global-to-local modeling to autoregressive transformers to mitigate the inference bottlenecks associated with self-attention. We demonstrate that Block Transformers reach 10--20x inference throughput compared to vanilla transformers with equivalent perplexity and zero-shot task performance.
  arXiv  Detail & Related papers  (2024-06-04T17:45:26Z)
• Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers [13.355338760884583]
  We propose Memory-directional Bi-efficient Transformer (MeBT) for end-to-end learning of long-term dependency in videos. Our method learns to decode entire-temporal volume of a video in parallel from partially observed patches.
  arXiv  Detail & Related papers  (2023-03-20T16:35:38Z)
• Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation [58.4650849317274]
  Volumetric Aggregation with Transformers (VAT) is a cost aggregation network for few-shot segmentation. VAT attains state-of-the-art performance for semantic correspondence as well, where cost aggregation also plays a central role.
  arXiv  Detail & Related papers  (2022-07-22T04:10:30Z)
• SepTr: Separable Transformer for Audio Spectrogram Processing [74.41172054754928]
  We propose a new vision transformer architecture called Separable Transformer (SepTr) SepTr employs two transformer blocks in a sequential manner, the first attending to tokens within the same frequency bin, and the second attending to tokens within the same time interval. We conduct experiments on three benchmark data sets, showing that our architecture outperforms conventional vision transformers and other state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-17T19:48:43Z)
• Vision Transformer with Progressive Sampling [73.60630716500154]
  We propose an iterative and progressive sampling strategy to locate discriminative regions. When trained from scratch on ImageNet, PS-ViT performs 3.8% higher than the vanilla ViT in terms of top-1 accuracy.
  arXiv  Detail & Related papers  (2021-08-03T18:04:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.