GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization

• URL: http://arxiv.org/abs/2601.22095v1
• Date: Thu, 29 Jan 2026 18:31:31 GMT
• Title: GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
• Abstract summary: We introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold.<n> Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models.<n>GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost.
• Score: 44.04664613316317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches through the lens of manifold optimization, interpreting the outputs of the Feed-Forward Network (FFN) and attention layers as update directions in optimization. Building on this perspective, we introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold. Furthermore, analogous to learning rate schedules, we propose a layer-wise update decay for the FFN and attention components. Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models. Crucially, GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost.

Related papers

• SpanNorm: Reconciling Training Stability and Performance in Deep Transformers [55.100133502295996]
  We propose SpanNorm, a novel technique designed to resolve the dilemma by integrating the strengths of both paradigms.<n>We provide a theoretical analysis demonstrating that SpanNorm, combined with a principled scaling strategy, maintains bounded signal variance throughout the network.<n> Empirically, SpanNorm consistently outperforms standard normalization schemes in both dense and Mixture-of-Experts (MoE) scenarios.
  arXiv  Detail & Related papers  (2026-01-30T05:21:57Z)
• A Constrained Optimization Perspective of Unrolled Transformers [77.12297732942095]
  We introduce a constrained optimization framework for training transformers that behave like optimization descent algorithms.<n>We observe constrained transformers achieve stronger to perturbations robustness and maintain higher out-of-distribution generalization.
  arXiv  Detail & Related papers  (2026-01-24T02:12:39Z)
• The Neural Differential Manifold: An Architecture with Explicit Geometric Structure [8.201374511929538]
  This paper introduces the Neural Differential Manifold (NDM), a novel neural network architecture that explicitly incorporates geometric structure into its fundamental design.<n>We analyze the theoretical advantages of this approach, including its potential for more efficient optimization, enhanced continual learning, and applications in scientific discovery and controllable generative modeling.
  arXiv  Detail & Related papers  (2025-10-29T02:24:27Z)
• GeoAda: Efficiently Finetune Geometric Diffusion Models with Equivariant Adapters [61.51810815162003]
  We propose an SE(3)-equivariant adapter framework ( GeoAda) that enables flexible and parameter-efficient fine-tuning for controlled generative tasks.<n>GeoAda preserves the model's geometric consistency while mitigating overfitting and catastrophic forgetting.<n>We demonstrate the wide applicability of GeoAda across diverse geometric control types, including frame control, global control, subgraph control, and a broad range of application domains.
  arXiv  Detail & Related papers  (2025-07-02T18:44:03Z)
• HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization [25.87557024380553]
  We propose a simple yet effective hybrid normalization strategy that integrates the advantages of Pre-Norm and Post-Norm.<n>In experiments on large-scale transformer models, we show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches.<n>These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models.
  arXiv  Detail & Related papers  (2025-03-06T16:40:48Z)
• ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models [3.7802450241986945]
  LayerNorm is a critical component in modern large language models (LLMs) for stabilizing training and ensuring smooth optimization. This work explores desirable activation functions in normalization-free decoder-only LLMs. ReLU significantly outperforms GELU in LayerNorm-free models, leading to an bf 8.2% perplexity improvement.
  arXiv  Detail & Related papers  (2024-10-12T20:26:01Z)
• GeoGaussian: Geometry-aware Gaussian Splatting for Scene Rendering [83.19049705653072]
  During the Gaussian Splatting optimization process, the scene's geometry can gradually deteriorate if its structure is not deliberately preserved. We propose a novel approach called GeoGaussian to mitigate this issue. Our proposed pipeline achieves state-of-the-art performance in novel view synthesis and geometric reconstruction.
  arXiv  Detail & Related papers  (2024-03-17T20:06:41Z)
• Full Stack Optimization of Transformer Inference: a Survey [58.55475772110702]
  Transformer models achieve superior accuracy across a wide range of applications. The amount of compute and bandwidth required for inference of recent Transformer models is growing at a significant rate. There has been an increased focus on making Transformer models more efficient.
  arXiv  Detail & Related papers  (2023-02-27T18:18:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.