Tapered Language Models

• URL: http://arxiv.org/abs/2606.23670v1
• Date: Mon, 22 Jun 2026 17:56:25 GMT
• Title: Tapered Language Models
• Abstract summary: Evidence suggests that layers contribute non-uniformly to the final output, with later refining the residual rather than transforming it.<n>Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts.
• Score: 17.586289013051864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This is a default inherited from the original transformer and largely unchanged since, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output, with later layers refining the residual stream rather than transforming it. We ask whether parameter capacity should reflect this asymmetry. Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts. Building on this result, we introduce Tapered Language Models (TLMs), an architectural principle in which a parameter-bearing component is monotonically tapered across depth under a fixed total budget. MLPs are the natural site for this instantiation: they dominate parameter count across all modern LM families and expose width as a single, clean axis of variation. Across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, and Titans), tapering MLP width via a smooth cosine schedule consistently improves perplexity and downstream benchmark performance over uniform baselines, at no additional parameter or compute cost. These findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of language model design, a free lever hidden in plain sight.

Related papers

• Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing [0.0]
  Multi-head attention partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension throughout the models depth.<n>We introduce the Prism Transformer, a novel architectural paradigm that replaces the static, uniform head configuration with a progressive head schedule.<n>By monotonically increasing the head count across layers, the Prism Transformer naturally establishes a local-to-global representational hierarchy.
  arXiv  Detail & Related papers  (2026-06-25T18:18:32Z)
• Variable-Width Transformers [57.07167443557886]
  We investigate nonuniform capacity allocation across network depth by proposing a $times$-shaped > former architecture.<n>Our design maintains wider early and late layers while narrowing the middle layers, utilizing a parameter-free residual resizing mechanism.<n>By reducing the average layer width, this architecture also requires fewer overall FLOPs and smaller KV cache memory and I/O cost.
  arXiv  Detail & Related papers  (2026-06-16T17:59:03Z)
• BA-T: An Iterative Transformer for Two-View Bundle Adjustment [53.16990349592508]
  We propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space.<n> experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations.<n>BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture.
  arXiv  Detail & Related papers  (2026-06-02T07:51:14Z)
• Attention Residuals [38.59138244826294]
  Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights.<n>We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs.
  arXiv  Detail & Related papers  (2026-03-16T09:32:21Z)
• Equivalence of Context and Parameter Updates in Modern Transformer Blocks [8.364690240329411]
  Recent research has established that the impact of context in a vanilla transformer can be represented implicitly by forming a token-dependent, rank-1 patch to its weights.<n>We first demonstrate a precise, analytical solution for a Gemma-style transformer block, proving that the entire effect of a context can be perfectly mapped to rank-1 patches.<n>We then generalize this result, providing a constructive proof and algorithm for multi-layer models.
  arXiv  Detail & Related papers  (2025-11-22T01:17:15Z)
• Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training [0.0]
  Transformer-based language models traditionally use uniform (isotropic) layer sizes, yet they ignore the diverse functional roles that different depths can play and their computational capacity needs.<n>We introduce three new literature variants - Framed, Reverse, and Crown - that redistribute FFN widths and attention heads via two or three-point linear in the pre-training stage.<n>We present the first systematic ablation of LWS and its variants, on a fixed budget of 180M parameters, trained on 5B tokens.<n>All models converge to similar losses and achieve better performance compared to an equal-cost isotropic baseline, without a substantial decrease in training
  arXiv  Detail & Related papers  (2025-09-08T10:24:19Z)
• You Do Not Fully Utilize Transformer's Representation Capacity [4.753535328327317]
  Layer-Integrated Memory (LIMe) is a lightweight extension that learns per-head, per-layer routing weights to integrate representations from all previous layers with negligible overhead.<n>LIMe consistently achieves faster convergence, lower perplexity per FLOP, and substantial accuracy improvements on synthetic tasks.
  arXiv  Detail & Related papers  (2025-02-13T12:00:50Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture [68.13678918660872]
  We design a more capable parameter-sharing architecture based on matrix product operator (MPO) MPO decomposition can reorganize and factorize the information of a parameter matrix into two parts. Our architecture shares the central tensor across all layers for reducing the model size.
  arXiv  Detail & Related papers  (2023-03-27T02:34:09Z)
• Equivariant Architectures for Learning in Deep Weight Spaces [54.61765488960555]
  We present a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trainedvariant. We show how these layers can be implemented using three basic operations.
  arXiv  Detail & Related papers  (2023-01-30T10:50:33Z)
• Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
  State-of-the-art neural language models (LMs) represented by Transformers are highly complex. This paper proposes a full Bayesian learning framework for Transformer LM estimation.
  arXiv  Detail & Related papers  (2021-02-09T10:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.