Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation

• URL: http://arxiv.org/abs/2305.09312v1
• Date: Tue, 16 May 2023 09:37:08 GMT
• Title: Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation
• Abstract summary: This paper studies the impact of layer normalization (LayerNorm) on zero-shot translation (ZST) Recent efforts for ZST often utilize the Transformer architecture as the backbone, with LayerNorm at the input of layers (PreNorm) set as the default.
• Score: 34.597300388169536
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper studies the impact of layer normalization (LayerNorm) on zero-shot translation (ZST). Recent efforts for ZST often utilize the Transformer architecture as the backbone, with LayerNorm at the input of layers (PreNorm) set as the default. However, Xu et al. (2019) has revealed that PreNorm carries the risk of overfitting the training data. Based on this, we hypothesize that PreNorm may overfit supervised directions and thus have low generalizability for ZST. Through experiments on OPUS, IWSLT, and Europarl datasets for 54 ZST directions, we demonstrate that the original Transformer setting of LayerNorm after residual connections (PostNorm) consistently outperforms PreNorm by up to 12.3 BLEU points. We then study the performance disparities by analyzing the differences in off-target rates and structural variations between PreNorm and PostNorm. This study highlights the need for careful consideration of the LayerNorm setting for ZST.

Related papers

• Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning [52.63618112418439]
  Sharpness-aware computation (SAM) seeks the minima with a flat loss landscape to improve the generalization performance in machine learning tasks, including fine-tuning.<n>We propose an approach SL-SAM to break this bottleneck by introducing the sparse technique to layers.
  arXiv  Detail & Related papers  (2026-02-10T04:05:43Z)
• SpanNorm: Reconciling Training Stability and Performance in Deep Transformers [55.100133502295996]
  We propose SpanNorm, a novel technique designed to resolve the dilemma by integrating the strengths of both paradigms.<n>We provide a theoretical analysis demonstrating that SpanNorm, combined with a principled scaling strategy, maintains bounded signal variance throughout the network.<n> Empirically, SpanNorm consistently outperforms standard normalization schemes in both dense and Mixture-of-Experts (MoE) scenarios.
  arXiv  Detail & Related papers  (2026-01-30T05:21:57Z)
• GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization [44.04664613316317]
  We introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold.<n> Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models.<n>GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost.
  arXiv  Detail & Related papers  (2026-01-29T18:31:31Z)
• A Constrained Optimization Perspective of Unrolled Transformers [77.12297732942095]
  We introduce a constrained optimization framework for training transformers that behave like optimization descent algorithms.<n>We observe constrained transformers achieve stronger to perturbations robustness and maintain higher out-of-distribution generalization.
  arXiv  Detail & Related papers  (2026-01-24T02:12:39Z)
• Impact of Layer Norm on Memorization and Generalization in Transformers [9.069887212265087]
  We investigate how LayerNorm influences memorization and learning for Pre- and Post-LayerNorm transformers.<n>We identify that LayerNorm serves as a key factor for stable learning in Pre-LayerNorm transformers, while in Post-LayerNorm transformers, it impacts memorization.<n>Our analysis reveals that eliminating LayerNorm parameters in Pre-LayerNorm models exacerbates memorization and destabilizes learning, while in Post-LayerNorm models, it effectively mitigates memorization by restoring genuine labels.
  arXiv  Detail & Related papers  (2025-11-13T18:07:07Z)
• Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification [30.16664767564679]
  We show that shifts in LayerNorm parameters after fine-tuning are indicative of the transitions between source and target domains.<n>Our findings illuminate the underexplored dynamics of LayerNorm in transfer learning and provide practical strategies for LayerNorm fine-tuning.
  arXiv  Detail & Related papers  (2025-08-11T03:18:47Z)
• HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization [25.87557024380553]
  We propose a straightforward yet effective hybrid normalization strategy that integrates the advantages of both Pre-Norm and Post-Norm approaches. Tests show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches. These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models.
  arXiv  Detail & Related papers  (2025-03-06T16:40:48Z)
• The Curse of Depth in Large Language Models [28.37870372690079]
  We introduce the Curse of Depth, a concept that highlights, explains, and addresses the recent observation in modern Large Language Models (LLMs) We first confirm the wide existence of this phenomenon across the most popular families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our experimental results, spanning model sizes from 130M to 1B, demonstrate that LayerNorm Scaling significantly enhances LLM pre-training performance compared to Pre-LN.
  arXiv  Detail & Related papers  (2025-02-09T07:03:36Z)
• ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models [3.7802450241986945]
  LayerNorm is a critical component in modern large language models (LLMs) for stabilizing training and ensuring smooth optimization. This work explores desirable activation functions in normalization-free decoder-only LLMs. ReLU significantly outperforms GELU in LayerNorm-free models, leading to an bf 8.2% perplexity improvement.
  arXiv  Detail & Related papers  (2024-10-12T20:26:01Z)
• A Layer Selection Approach to Test Time Adaptation [0.2968738145616401]
  Test Time Adaptation (TTA) addresses the problem of distribution shift by adapting a pretrained model to a new domain during inference. We propose GALA, a novel layer selection criterion to identify the most beneficial updates to perform during TTA.
  arXiv  Detail & Related papers  (2024-04-04T19:55:11Z)
• CR-SAM: Curvature Regularized Sharpness-Aware Minimization [8.248964912483912]
  Sharpness-Aware Minimization (SAM) aims to enhance the generalizability by minimizing worst-case loss using one-step gradient ascent as an approximation. In this paper, we introduce a normalized Hessian trace to accurately measure the curvature of loss landscape on em both training and test sets. In particular, to counter excessive non-linearity of loss landscape, we propose Curvature Regularized SAM (CR-SAM)
  arXiv  Detail & Related papers  (2023-12-21T03:46:29Z)
• Landslide Surface Displacement Prediction Based on VSXC-LSTM Algorithm [11.282712030154569]
  We conduct modelling research on real unidirectional surface displacement data of recent landslides in the research area. We propose a time series prediction framework named VMD-SegSigmoid-XGBoost-ClusterLSTM based on variational mode decomposition.
  arXiv  Detail & Related papers  (2023-07-24T04:46:22Z)
• Normalization Layers Are All That Sharpness-Aware Minimization Needs [53.799769473526275]
  Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima. We show that perturbing only the affine normalization parameters (typically comprising 0.1% of the total parameters) in the adversarial step of SAM can outperform perturbing all of the parameters.
  arXiv  Detail & Related papers  (2023-06-07T08:05:46Z)
• Sharpness-Aware Training for Free [163.1248341911413]
  SharpnessAware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. Sharpness-Aware Training Free (SAF) mitigates the sharp landscape at almost zero computational cost over the base. SAF ensures the convergence to a flat minimum with improved capabilities.
  arXiv  Detail & Related papers  (2022-05-27T16:32:43Z)
• Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs? [0.0]
  We use a well-performing text generative adversarial network (GAN) architecture - Diversity-Promoting GAN (DPGAN) We attempted a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer in order to leverage their efficiency. The resulting Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality and diversity of generated text and stability.
  arXiv  Detail & Related papers  (2021-08-26T14:15:36Z)
• Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior [54.629850694790036]
  spectral-normalized identity priors (SNIP) is a structured pruning approach that penalizes an entire residual module in a Transformer model toward an identity mapping. We conduct experiments with BERT on 5 GLUE benchmark tasks to demonstrate that SNIP achieves effective pruning results while maintaining comparable performance.
  arXiv  Detail & Related papers  (2020-10-05T05:40:56Z)
• Learned Multi-layer Residual Sparsifying Transform Model for Low-dose CT Reconstruction [11.470070927586017]
  Sparsifying transform learning involves highly efficient sparse coding and operator update steps. We propose a Multi-layer Residual Sparsifying Transform (MRST) learning model wherein the transform domain residuals are jointly sparsified over layers.
  arXiv  Detail & Related papers  (2020-05-08T02:36:50Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)
• On Layer Normalization in the Transformer Architecture [112.40350994368741]
  We first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. We show in experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines.
  arXiv  Detail & Related papers  (2020-02-12T00:33:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.