Gated Removal of Normalization in Transformers Enables Stable Training and Efficient Inference

• URL: http://arxiv.org/abs/2602.10408v1
• Date: Wed, 11 Feb 2026 01:40:34 GMT
• Title: Gated Removal of Normalization in Transformers Enables Stable Training and Efficient Inference
• Abstract summary: We introduce TaperNorm, a drop-in replacement for RMSNorm/LayerNorm that behaves exactly like the standard normalizer early in training.<n>A single global gate is held at $g=1$ during gate warmup, used to calibrate the scaling branch via EMAs, and then cosine-decayed to $g=0$.<n>On an efficiency microbenchmark, folding internal scalings yields up to $1.22times$ higher throughput in last-token logits mode.
• Score: 2.1665689529884697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Normalization is widely viewed as essential for stabilizing Transformer training. We revisit this assumption for pre-norm Transformers and ask to what extent sample-dependent normalization is needed inside Transformer blocks. We introduce TaperNorm, a drop-in replacement for RMSNorm/LayerNorm that behaves exactly like the standard normalizer early in training and then smoothly tapers to a learned sample-independent linear/affine map. A single global gate is held at $g{=}1$ during gate warmup, used to calibrate the scaling branch via EMAs, and then cosine-decayed to $g{=}0$, at which point per-token statistics vanish and the resulting fixed scalings can be folded into adjacent linear projections. Our theoretical and empirical results isolate scale anchoring as the key role played by output normalization: as a (near) $0$-homogeneous map it removes radial gradients at the output, whereas without such an anchor cross-entropy encourages unbounded logit growth (``logit chasing''). We further show that a simple fixed-target auxiliary loss on the pre-logit residual-stream scale provides an explicit alternative anchor and can aid removal of the final normalization layer. Empirically, TaperNorm matches normalized baselines under identical setups while eliminating per-token statistics and enabling these layers to be folded into adjacent linear projections at inference. On an efficiency microbenchmark, folding internal scalings yields up to $1.22\times$ higher throughput in last-token logits mode. These results take a step towards norm-free Transformers while identifying the special role output normalization plays.

Related papers

• Dynamical phase selection controls compute scaling in looped transformers [0.0]
  We show that networks with identical architecture and objective, trained to identical accuracy, nevertheless realize distinct dynamical phases.<n>The phases are distinguished by their bifurcation mechanisms, including a saddle-node fold and a Neimark-Sacker-type transition to bounded nonstationary motion.
  arXiv  Detail & Related papers  (2026-08-27T02:54:02Z)
• Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation [3.0459410574367314]
  We introduce Mean Root Square Normalization (MRSNorm)<n>MRSNorm mathematically inverts the traditional scaling paradigm.<n>We analytically demonstrate that this constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity.
  arXiv  Detail & Related papers  (2026-07-20T11:10:51Z)
• Direct/adaptive-mixture phase-gradient learning for neural-network quantum states with complex phase structure [3.1413021865737765]
  Estimator design emerges as a first-class lever for complex neural quantum states.<n>We show how a small network trained this way reaches $0.89%$ median error, where tuned standard baselines plateau at $1.8%$.<n>The advantage carries over to chiral XXX chains: the direct estimator again converges to a markedly lower error than the standard one, across $$ and size.
  arXiv  Detail & Related papers  (2026-06-11T21:05:28Z)
• An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers [0.0]
  Transformers consuming multi-channel scalar signals must embed $C$ simultaneous values into one $d_textmodel$-dimensional vector per time step.<n>We audit eight input encoders -- a shared-scalar baseline, per-channel linear projections, an explicit regulariser, a nonlinearoretic, block-partitioned concatenation, channel-independent and channel-as-token.<n>We find that the standard per-channel linear projection matches every alternative up to small, statistically real but practically modest differences.
  arXiv  Detail & Related papers  (2026-06-03T11:35:09Z)
• N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation [68.47358899451255]
  N-vium is a mixture-of-exits transformer that partially parallelizes computation across depth on standard hardware.<n>N-vium attaches prediction heads at multiple depths and defines the next-token distribution as a learned mixture over these exits.
  arXiv  Detail & Related papers  (2026-05-13T08:46:17Z)
• Distributional Spectral Diagnostics for Localizing Grokking Transitions [4.480801209867367]
  In grokking, a model first fits the training data while test accuracy remains low, and only later begins to generalize.<n>We ask whether this transition can be localized from observed training trajectories before the test accuracy rises, and formulate grokking transition localization as a diagnostic problem with an explicit threshold/FPR/lead-time trade-off.
  arXiv  Detail & Related papers  (2026-05-07T03:08:58Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• Transformer Normalisation Layers and the Independence of Semantic Subspaces [17.957364289876548]
  We consider a semantic subspace to be any independent subspace of the latent representation that can fully determine an attention distribution. We show that Pre-Norm, the placement of normalisation layer used by state-of-the-art transformers, violates this ability. We observe a 1% rate of circuit collapse when the norms are artificially perturbed by $lesssim$10%.
  arXiv  Detail & Related papers  (2024-06-25T16:16:38Z)
• Implicit Bias and Fast Convergence Rates for Self-attention [26.766649949420746]
  We study the fundamental optimization principles of self-attention, the defining mechanism of transformers.<n>We analyze the implicit bias of gradient-baseds in a self-attention layer with a decoder in a linear classification.
  arXiv  Detail & Related papers  (2024-02-08T15:15:09Z)
• p-Laplacian Transformer [7.2541371193810384]
  $p$-Laplacian regularization, rooted in graph and image signal processing, introduces a parameter $p$ to control the regularization effect on these data. We first show that the self-attention mechanism obtains the minimal Laplacian regularization. We then propose a novel class of transformers, namely the $p$-Laplacian Transformer (p-LaT)
  arXiv  Detail & Related papers  (2023-11-06T16:25:56Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks [117.93273337740442]
  We show that gradient descent converges to a uniform margin classifier on the training data with an $exp(-Omega(log2 t))$ convergence rate. We also show that batch normalization has an implicit bias towards a patch-wise uniform margin.
  arXiv  Detail & Related papers  (2023-06-20T16:58:00Z)
• Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation [58.4650849317274]
  Volumetric Aggregation with Transformers (VAT) is a cost aggregation network for few-shot segmentation. VAT attains state-of-the-art performance for semantic correspondence as well, where cost aggregation also plays a central role.
  arXiv  Detail & Related papers  (2022-07-22T04:10:30Z)
• Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior [54.629850694790036]
  spectral-normalized identity priors (SNIP) is a structured pruning approach that penalizes an entire residual module in a Transformer model toward an identity mapping. We conduct experiments with BERT on 5 GLUE benchmark tasks to demonstrate that SNIP achieves effective pruning results while maintaining comparable performance.
  arXiv  Detail & Related papers  (2020-10-05T05:40:56Z)
• On Layer Normalization in the Transformer Architecture [112.40350994368741]
  We first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. We show in experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines.
  arXiv  Detail & Related papers  (2020-02-12T00:33:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.