Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

• URL: http://arxiv.org/abs/2607.08186v1
• Date: Thu, 09 Jul 2026 07:37:59 GMT
• Title: Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
• Abstract summary: We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.<n>We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining.
• Score: 75.80275843320511
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed. Depth-recurrent (looped) Transformers pursue this goal but are hard to scale, because looped computation does not fit naturally with the pipeline parallelism used to train the largest models. We add computation along the sequence-length dimension, where the extra computation is simply a longer input and stays compatible with standard large-model training. We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining (CPT). It expands each token into n streams with independent embedding tables and keeps the intermediate streams' key-value cache as context, so each token performs more internal computation without adding or widening Transformer layers. To keep this affordable at scale, we introduce Stream-Factorized Attention, in which most layers attend only within each stream and only a few layers mix across streams, reducing the attention cost from quadratic to roughly linear in n. Experiments support two scaling results. At frontier scale, we train WeLM-HD4-80B and WeLM-HD4-617B at n=4 and improve their matched non-HD baselines, making Hidden Decoding the first demonstrated sequence-length scaling method at the 100B+ MoE scale. Across expansion factors, the gains grow as n increases, showing that sequence-length expansion is a practical fixed-backbone scaling path for frontier-scale LLMs.

Related papers

• T^2MLR: Transformer with Temporal Middle-Layer Recurrence [28.818755729513793]
  We develop a transformers-based latent reasoning architecture that fuses a cached middle layer representation from the previous token directly into an earlier layer of the current token position.<n>Applying recurrence to only a localized middle-layer block (as little as 20% of the network) often outperforms full-layer recurrence.<n>These results suggest that effective latent reasoning in Transformers does not require looping over all layers as in previous works, but can instead emerge more strongly from targeted middle-layer recurrence.
  arXiv  Detail & Related papers  (2026-07-16T16:33:59Z)
• Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling [9.458812631271009]
  We introduce Nexusformer, which replaces linear $Q/K/V projections with a Nexus-Rank layer.<n>New capacity can be injected along two axes via zero-d blocks that preserve pretrained knowledge.<n>Experiments on language modeling and reasoning demonstrate that Nexusformer matches Tokenformer's perplexity benchmarks.
  arXiv  Detail & Related papers  (2026-04-21T06:54:16Z)
• From Projection to Prediction: Beyond Logits for Scalable Language Models [0.28647133890966986]
  Training Large Language Models (LLMs) typically involves a two-stage pipeline at the output layer.<n>By directly computing the loss from hidden states and target tokens, our approach bypasses explicit logits materialization.
  arXiv  Detail & Related papers  (2025-11-18T02:23:47Z)
• SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers [15.142822497807236]
  We propose SCOUT, a hybrid architecture that compresses tokens locally within fixed-size segments and applies attention only over these compressed representations.<n>SCOUT retains much of the expressivity of full attention while substantially reducing the computational and memory cost.<n>We analyze SCOUT's computational and memory efficiency and evaluate it empirically on long-context language modeling and reasoning tasks.
  arXiv  Detail & Related papers  (2025-08-31T17:08:33Z)
• StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMs [8.960494482210919]
  We propose a memory-efficient Backpropagation (BP) method called StreamBP.<n>StreamBP performs a linear decomposition of the chain rule along the sequence dimension in a layer-wise manner.<n>Compared to gradient checkpointing, StreamBP scales up the maximum sequence length of BP by 2.8-5.5 times larger.
  arXiv  Detail & Related papers  (2025-06-03T16:54:15Z)
• LESA: Learnable LLM Layer Scaling-Up [57.0510934286449]
  Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive.<n>Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones.<n>We propose textbfLESA, a novel learnable method for depth scaling-up.
  arXiv  Detail & Related papers  (2025-02-19T14:58:48Z)
• Byte Latent Transformer: Patches Scale Better Than Tokens [101.10994909832063]
  Byte Latent Transformer (BLT) encodes bytes into dynamically sized patches, which serve as the primary units of computation.<n>For fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.
  arXiv  Detail & Related papers  (2024-12-13T05:33:32Z)
• DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models [34.74093040678323]
  We introduce DeepSpeed-Ulysses, a novel, portable and effective methodology for enabling highly efficient and scalable LLM training. DeepSpeed-Ulysses at its core partitions input data along the sequence dimension and employs an efficient all-to-all collective communication for attention. Experiments show that DeepSpeed-Ulysses trains 2.5x faster with 4x longer sequence length than the existing method SOTA baseline.
  arXiv  Detail & Related papers  (2023-09-25T20:15:57Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• CloudAttention: Efficient Multi-Scale Attention Scheme For 3D Point Cloud Learning [81.85951026033787]
  We set transformers in this work and incorporate them into a hierarchical framework for shape classification and part and scene segmentation. We also compute efficient and dynamic global cross attentions by leveraging sampling and grouping at each iteration. The proposed hierarchical model achieves state-of-the-art shape classification in mean accuracy and yields results on par with the previous segmentation methods.
  arXiv  Detail & Related papers  (2022-07-31T21:39:15Z)
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing [112.2208052057002]
  We propose Funnel-Transformer which gradually compresses the sequence of hidden states to a shorter one. With comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on a wide variety of sequence-level prediction tasks.
  arXiv  Detail & Related papers  (2020-06-05T05:16:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.