The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?

• URL: http://arxiv.org/abs/2605.20749v2
• Date: Mon, 25 May 2026 12:28:18 GMT
• Title: The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
• Abstract summary: Gated Linear Units (GLU) and their variants are widely adopted in modern open-source large language model architectures.<n>We study GLU by analyzing two-layer networks in the neural tangent kernel (NTK) regime.<n>Our analysis reveals that the GLU structure reshapes the NTK spectrum, leading to a smaller condition number and a more compact eigenvalue distribution.
• Score: 87.93926438258232
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gated Linear Units (GLU) and their variants are widely adopted in modern open-source large language model architectures and consistently outperform their non-gated counterparts, yet the underlying reasons for this advantage remain unclear. In this work, we study GLU by analyzing two-layer networks in the neural tangent kernel (NTK) regime. Our analysis reveals that the GLU structure reshapes the NTK spectrum, leading to a smaller condition number and a more compact eigenvalue distribution. Building on this finding, we further analyze the resulting training dynamics and show how the reshaped spectrum leads to faster convergence of GLU models, including a characteristic loss-crossing phenomenon observed between GLU and non-GLU models. Finally, we empirically observe that GLU has limited impact in reducing the generalization gap on various models, including ViT and GPT-2, suggesting that its primary benefit lies in accelerating optimization rather than reducing the generalization gap. The code is available at: https://github.com/Zemdalk/GLU-NTK.

Related papers

• λ-GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks [1.1145952934885128]
  Gaussian Error Linear Unit (GELU) is a widely used smooth alternative to Rectifier Linear Unit (ReLU)<n>-GELU provides a minimal and interpretable knob to profile and control gating hardness.<n>Overall, -GELU provides a minimal and interpretable knob to profile and control gating hardness, bridging smooth training with ReLU-centric downstream pipelines.
  arXiv  Detail & Related papers  (2026-03-23T13:58:19Z)
• IGLU: The Integrated Gaussian Linear Unit Activation Function [13.305282275999778]
  We introduce IGLU, a parametric activation function derived as a scale mixture of GELU gates under a half-normal mixing distribution.<n>We show that IGLU achieves competitive or superior performance on both vision and language datasets against ReLU and GELU baselines.
  arXiv  Detail & Related papers  (2026-03-06T20:28:08Z)
• L2G-Net: Local to Global Spectral Graph Neural Networks via Cauchy Factorizations [31.210700220124192]
  We propose a new class of spectral graph neural networks (GNNs) based on a factorization of the graph Fourier transform (GFT)<n>We show that L2G-Net outperforms existing spectral techniques and is competitive with the state-of-the-art with orders of magnitude fewer learnable parameters.
  arXiv  Detail & Related papers  (2026-02-21T13:53:36Z)
• Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations [53.348574336527854]
  We investigate the neural network map as a function on the space of weights and biases.<n>We prove, for the first time, the non-singularity of the gradient descent (GD) map on the loss landscape of realistic neural network architectures.
  arXiv  Detail & Related papers  (2025-10-28T14:34:33Z)
• A Generative Graph Contrastive Learning Model with Global Signal [0.0]
  Contrastive Signal Generative Framework for Accurate Graph Learning (CSG2L)<n>This study proposes a novel Contrastive Signal Generative Framework for Accurate Graph Learning (CSG2L)<n>Experiments on benchmark datasets demonstrate that the proposed CSG2L outperforms the state-of-art baselines.
  arXiv  Detail & Related papers  (2025-04-25T08:00:38Z)
• On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks [56.78271181959529]
  Kolmogorov--Arnold Networks (KANs) have gained significant attention in the deep learning community. Empirical investigations demonstrate that KANs optimized via gradient descent (SGD) are capable of achieving near-zero training loss.
  arXiv  Detail & Related papers  (2024-10-10T15:34:10Z)
• Expressivity and Generalization: Fragment-Biases for Molecular GNNs [42.64483757766247]
  We propose the Fragment-WL test, an extension to the well-known Weisfeiler & Leman test, which enables the theoretic analysis of fragment-biased GNNs. We develop a new GNN architecture and a fragmentation with infinite vocabulary that significantly boosts expressiveness. We show that our model exhibits superior generalization capabilities compared to the latest transformer-based architectures.
  arXiv  Detail & Related papers  (2024-06-12T13:41:07Z)
• Causal Inference in Gene Regulatory Networks with GFlowNet: Towards Scalability in Large Systems [87.45270862120866]
  We introduce Swift-DynGFN as a novel framework that enhances causal structure learning in GRNs. Specifically, Swift-DynGFN exploits gene-wise independence to boost parallelization and to lower computational cost.
  arXiv  Detail & Related papers  (2023-10-05T14:59:19Z)
• Fixing the NTK: From Neural Network Linearizations to Exact Convex Programs [63.768739279562105]
  We show that for a particular choice of mask weights that do not depend on the learning targets, this kernel is equivalent to the NTK of the gated ReLU network on the training data. A consequence of this lack of dependence on the targets is that the NTK cannot perform better than the optimal MKL kernel on the training set.
  arXiv  Detail & Related papers  (2023-09-26T17:42:52Z)
• From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent [50.4531316289086]
  Gradient Descent (SGD) has been the method of choice for learning large-scale non-root models. An overarching paper is providing general conditions SGD converges, assuming that GF on the population loss converges. We provide a unified analysis for GD/SGD not only for classical settings like convex losses, but also for more complex problems including Retrieval Matrix sq-root.
  arXiv  Detail & Related papers  (2022-10-13T03:55:04Z)
• Uncovering the Structural Fairness in Graph Contrastive Learning [87.65091052291544]
  Graph contrastive learning (GCL) has emerged as a promising self-supervised approach for learning node representations. We show that representations obtained by GCL methods are already fairer to degree bias than those learned by GCN. We devise a novel graph augmentation method, called GRAph contrastive learning for DEgree bias (GRADE), which applies different strategies to low- and high-degree nodes.
  arXiv  Detail & Related papers  (2022-10-06T15:58:25Z)
• Local SGD Optimizes Overparameterized Neural Networks in Polynomial Time [15.04034188283642]
  Local (S)GD (or FedAvg) can optimize two-layer neural networks with Rectified Linear Unit (ReLU) activation function. We show that, even though ReLU network does not admit gradient Lipschitzness property, the difference between gradients on local models and average model will not change too much.
  arXiv  Detail & Related papers  (2021-07-22T18:06:33Z)
• Optimization and Generalization Analysis of Transduction through Gradient Boosting and Application to Multi-scale Graph Neural Networks [60.22494363676747]
  It is known that the current graph neural networks (GNNs) are difficult to make themselves deep due to the problem known as over-smoothing. Multi-scale GNNs are a promising approach for mitigating the over-smoothing problem. We derive the optimization and generalization guarantees of transductive learning algorithms that include multi-scale GNNs.
  arXiv  Detail & Related papers  (2020-06-15T17:06:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.