NormFormer: Improved Transformer Pretraining with Extra Normalization

• URL: http://arxiv.org/abs/2110.09456v1
• Date: Mon, 18 Oct 2021 16:47:45 GMT
• Title: NormFormer: Improved Transformer Pretraining with Extra Normalization
• Abstract summary: NormFormer architecture adds three normalization operations to each layer. Extra operations incur negligible compute cost. For masked language modeling, NormFormer improves fine-tuned GLUE performance by 1.9%.
• Score: 31.250781670447257
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be alleviated by our proposed NormFormer architecture, which adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first fully connected layer. The extra operations incur negligible compute cost (+0.4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2.7 Billion parameters. For example, adding NormFormer on top of our strongest 1.3B parameter baseline can reach equal perplexity 24% faster, or converge 0.27 perplexity better in the same compute budget. This model reaches GPT3-Large (1.3B) zero shot performance 60% faster. For masked language modeling, NormFormer improves fine-tuned GLUE performance by 1.9% on average. Code to train NormFormer models is available in fairseq https://github.com/pytorch/fairseq/tree/main/examples/normformer .

Related papers

• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models [75.80275843320511]
  We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.<n>We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining.
  arXiv  Detail & Related papers  (2026-07-09T07:37:59Z)
• OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling [10.699320611591332]
  We analyze why three natural Muon-LAMB hybrids fail through shape-degenerates, raw-momentum clip saturation, de weight-decay runaway, and show that the decay avoids these failures.<n>We introduce OrScale, a trust-valued layer measure of Muon built on a simple rule.
  arXiv  Detail & Related papers  (2026-05-08T14:47:56Z)
• AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training [0.2578242050187029]
  AdamHuberDecay is a drop-in replacement for AdamW that substitutes the $ell$ penalty with a decoupled smooth Huber regularizer.<n>Experiments on GPT-2 and GPT-3 pre-training demonstrate that AdamHuberDecay converges 10-15% faster in wall-clock time.
  arXiv  Detail & Related papers  (2025-11-18T18:08:20Z)
• Diffusion Language Models are Super Data Learners [61.721441061210896]
  When unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs.<n>We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation.
  arXiv  Detail & Related papers  (2025-11-05T08:17:42Z)
• Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Models [80.50996301430108]
  sparse tuning achieves remarkable performance by adjusting only the weights most relevant to downstream tasks.<n>We propose a one-stage method named SNELLA to overcome the above limitations.<n> SNELLA achieves SOTA performance with low memory usage.
  arXiv  Detail & Related papers  (2025-10-28T03:39:18Z)
• Optimal Scaling Needs Optimal Norm [1.8180584498244492]
  Joint optimal scaling across model and dataset sizes is governed by a single invariant.<n>Across models with up to 1.3B parameters trained on up to 138B tokens, the optimal learning rate/batch size pair $(etaast, Bast)$ consistently has the same operator norm value.
  arXiv  Detail & Related papers  (2025-10-04T16:48:36Z)
• AdaGC: Improving Training Stability for Large Language Model Pretraining [18.163318397205533]
  Large LanguageText Models (LLMs) face increasing loss spikes during scaling. While global clipping mitigates this, traditional approaches mitigate specific variations. We show that AdaGC converges 25% faster than global clipping.
  arXiv  Detail & Related papers  (2025-02-16T08:13:23Z)
• Training Deep Learning Models with Norm-Constrained LMOs [56.00317694850397]
  We propose a new family of algorithms that uses the linear minimization oracle (LMO) to adapt to the geometry of the problem.<n>We demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam.
  arXiv  Detail & Related papers  (2025-02-11T13:10:34Z)
• Methods of improving LLM training stability [3.3790712267402374]
  We observe that with a high learning rate the L2 norm of all linear layer outputs can grow with each training step and the model diverges. We show that with the last two methods we can increase learning rate by 1.5x in comparison to an approach based on QK layer normalization only.
  arXiv  Detail & Related papers  (2024-10-22T04:27:03Z)
• Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes [18.12700358872208]
  Non-uniformity of the norm of the parameters is one of the causes of loss spikes. In training of neural networks, the scale of the gradients is required to be kept constant throughout the layers.
  arXiv  Detail & Related papers  (2024-10-07T14:09:58Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)
• Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost [53.746169882193456]
  Recent works have proposed various sparse attention modules to overcome the quadratic cost of self-attention. We propose a model that resolves both problems by endowing each attention head with a mixed-membership Block Model. Our model outperforms previous efficient variants as well as the original Transformer with full attention.
  arXiv  Detail & Related papers  (2022-10-27T15:30:52Z)
• AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs [23.523389372182613]
  gradient descent (SGD)s are generally used to train the convolutional neural networks (CNNs) Existing SGDs do not exploit the gradient norm of past iterations and lead to poor convergence and performance. We propose a novel AdaNorm based SGDs by correcting the norm of gradient in each iteration based on the adaptive training history of gradient norm.
  arXiv  Detail & Related papers  (2022-10-12T16:17:25Z)
• DeepNet: Scaling Transformers to 1,000 Layers [106.33669415337135]
  We introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer. In-depth theoretical analysis shows that model updates can be bounded in a stable way. We successfully scale Transformers up to 1,000 layers without difficulty, which is one order of magnitude deeper than previous deep Transformers.
  arXiv  Detail & Related papers  (2022-03-01T15:36:38Z)
• Robust Training of Neural Networks using Scale Invariant Architectures [70.67803417918854]
  In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks. We show that this general approach is robust to rescaling of parameter and loss. We design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam.
  arXiv  Detail & Related papers  (2022-02-02T11:58:56Z)
• Primer: Searching for Efficient Transformers for Language Modeling [79.2677566332444]
  Training and inference costs of large Transformer models have grown rapidly and become expensive. Here we aim to reduce the costs of Transformers by searching for a more efficient variant. We identify an architecture, named Primer, that has a smaller training cost than the original Transformer.
  arXiv  Detail & Related papers  (2021-09-17T17:50:39Z)
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention [119.77305080520718]
  We propose a new model architecture DeBERTa that improves the BERT and RoBERTa models using two novel techniques. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding (NLU) and natural langauge generation (NLG) downstream tasks.
  arXiv  Detail & Related papers  (2020-06-05T19:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.