IGLU: The Integrated Gaussian Linear Unit Activation Function

• URL: http://arxiv.org/abs/2603.06861v1
• Date: Fri, 06 Mar 2026 20:28:08 GMT
• Title: IGLU: The Integrated Gaussian Linear Unit Activation Function
• Abstract summary: We introduce IGLU, a parametric activation function derived as a scale mixture of GELU gates under a half-normal mixing distribution.<n>We show that IGLU achieves competitive or superior performance on both vision and language datasets against ReLU and GELU baselines.
• Score: 13.305282275999778
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Activation functions are fundamental to deep neural networks, governing gradient flow, optimization stability, and representational capacity. Within historic deep architectures, while ReLU has been the dominant choice for the activation function, modern transformer-based models increasingly are adopting smoother alternatives such as GELU and other self-gated alternatives. Despite their empirical success, the mathematical relationships among these functions and the principles underlying their effectiveness remains only partially understood. We introduce IGLU, a parametric activation function derived as a scale mixture of GELU gates under a half-normal mixing distribution. This derivation yields a closed-form expression whose gating component is exactly the Cauchy CDF, providing a principled one-parameter family that continuously interpolates between identity-like and ReLU-like behavior via a single sharpness parameter $σ$. Unlike GELU's Gaussian gate, IGLU's heavy-tailed Cauchy gate decays polynomially in the negative tail, guaranteeing non-zero gradients for all finite inputs and offering greater robustness to vanishing gradients. We further introduce IGLU-Approx, a computationally efficient rational approximation of IGLU expressed entirely in terms of ReLU operations that eliminates transcendental function evaluation. Through evaluations on CIFAR-10, CIFAR-100, and WikiText-103 across ResNet-20, ViT-Tiny, and GPT-2 Small, IGLU achieves competitive or superior performance on both vision and language datasets against ReLU and GELU baselines, with IGLU-Approx recovering this performance at substantially reduced computational cost. In particular, we show that employing a heavy-tailed gate leads to considerable performance gains in heavily imbalanced classification datasets.

Related papers

• The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure? [87.93926438258232]
  Gated Linear Units (GLU) and their variants are widely adopted in modern open-source large language model architectures.<n>We study GLU by analyzing two-layer networks in the neural tangent kernel (NTK) regime.<n>Our analysis reveals that the GLU structure reshapes the NTK spectrum, leading to a smaller condition number and a more compact eigenvalue distribution.
  arXiv  Detail & Related papers  (2026-05-20T05:50:06Z)
• Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes [53.726365933748134]
  We analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions.<n>We characterize the sparsity-accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above $sim$70% activation sparsity.
  arXiv  Detail & Related papers  (2026-05-17T21:29:20Z)
• Adaptive Threshold-Driven Continuous Greedy Method for Scalable Submodular Optimization [0.0]
  Submodular under matroid constraints is a fundamental problem in optimization with applications in sensing, data-marization, active learning, and resource allocation.<n>We propose textitATCG (underlineAdaptive underlineThresholded underlineThresholded underlineGreedy), expanding each agent's active set only when current candidates fail to capture sufficient marginal gain.<n>Experiments on a classbalanced prototype selection problem over a CIFAR-10 animal dataset show that textit
  arXiv  Detail & Related papers  (2026-04-03T19:32:39Z)
• Shapley-Guided Neural Repair Approach via Derivative-Free Optimization [21.274208005071277]
  Existing approaches mainly involve retraining, optimization, constraint, or search algorithms.<n>We propose SHARPEN, integrating interpretable fault localization with a derivative-free optimization strategy.
  arXiv  Detail & Related papers  (2026-04-01T03:05:48Z)
• λ-GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks [1.1145952934885128]
  Gaussian Error Linear Unit (GELU) is a widely used smooth alternative to Rectifier Linear Unit (ReLU)<n>-GELU provides a minimal and interpretable knob to profile and control gating hardness.<n>Overall, -GELU provides a minimal and interpretable knob to profile and control gating hardness, bridging smooth training with ReLU-centric downstream pipelines.
  arXiv  Detail & Related papers  (2026-03-23T13:58:19Z)
• Unbiased Gradient Estimation for Event Binning via Functional Backpropagation [64.88399635309918]
  We propose a novel framework for unbiased gradient estimation of arbitrary binning functions by synthesizing weak derivatives during backpropagation.<n>We achieve 9.4% lower EPE in self-supervised optical flow, and 5.1% lower RMS error in SLAM, demonstrating broad benefits for event-based visual perception.
  arXiv  Detail & Related papers  (2026-02-13T04:05:03Z)
• Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics [39.0860823332923]
  GoLU is a novel self-gated activation function defined as $mathrmGoLU(x) = x, mathrmGompertz(x)$, where $mathrmGompertz(x) = e-e-x$.<n>GoLU is superior to state-of-the-art activation functions, establishing GoLU as a robust alternative to existing activation functions.
  arXiv  Detail & Related papers  (2025-02-05T22:32:22Z)
• Deriving Activation Functions Using Integration [8.345753173238956]
  We introduce the Expanded Integral of the Exponential Linear Unit (xIELU), a trainable piecewise activation function derived by integrating trainable affine transformations.<n>xIELU combines two key properties for the gradient: (1) a trainable and linearly increasing gradient for positive inputs, similar to Squared ReLU (ReLU$2$), and (2) a trainable gradient that can take negative values for negative inputs, inspired by Expanded SiLU (xSiLU)<n>In experiments with 1.1B and 3B parameter Llama models trained on 125B tokens of FineWeb Edu, xIELU achieves lower
  arXiv  Detail & Related papers  (2024-11-20T03:24:21Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Graph-adaptive Rectified Linear Unit for Graph Neural Networks [64.92221119723048]
  Graph Neural Networks (GNNs) have achieved remarkable success by extending traditional convolution to learning on non-Euclidean data. We propose Graph-adaptive Rectified Linear Unit (GReLU) which is a new parametric activation function incorporating the neighborhood information in a novel and efficient way. We conduct comprehensive experiments to show that our plug-and-play GReLU method is efficient and effective given different GNN backbones and various downstream tasks.
  arXiv  Detail & Related papers  (2022-02-13T10:54:59Z)
• A Unified Off-Policy Evaluation Approach for General Value Function [131.45028999325797]
  General Value Function (GVF) is a powerful tool to represent both predictive and retrospective knowledge in reinforcement learning (RL) In this paper, we propose a new algorithm called GenTD for off-policy GVFs evaluation. We show that GenTD learns multiple interrelated multi-dimensional GVFs as efficiently as a single canonical scalar value function.
  arXiv  Detail & Related papers  (2021-07-06T16:20:34Z)
• Approximation Schemes for ReLU Regression [80.33702497406632]
  We consider the fundamental problem of ReLU regression. The goal is to output the best fitting ReLU with respect to square loss given to draws from some unknown distribution.
  arXiv  Detail & Related papers  (2020-05-26T16:26:17Z)
• Gaussian Error Linear Units (GELUs) [58.195342948092964]
  We propose a neural network activation function that weights inputs by their value, rather than gates by their sign. We find performance improvements across all considered computer vision, natural language processing, and speech tasks.
  arXiv  Detail & Related papers  (2016-06-27T19:20:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.