Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm

• URL: http://arxiv.org/abs/2606.15669v1
• Date: Sun, 14 Jun 2026 08:22:28 GMT
• Title: Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
• Abstract summary: Modern deep neural networks rely on Euclidean scalar activations (e.g., ReLU) to prevent gradient instability in deep architectures.<n>We propose the Z-Plane Neural Network, which maps hidden states into 2D phasor bundles on a hypersphere.<n>We demonstrate mathematically that this isotropic activation maintains 1-Lipschitz continuity and prevents gradient vanishing by preserving tangential gradients.
• Score: 3.0459410574367314
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern deep neural networks rely on Euclidean scalar activations (e.g., ReLU) and global normalization techniques (e.g., LayerNorm) to prevent gradient instability in deep architectures. However, these mechanisms inherently cause dead neurons, discard critical directional information, and destroy the orthogonality of feature representations. Inspired by the frequency-modulation transmission of biological axons, we propose the Z-Plane Neural Network, which maps hidden states into 2D phasor bundles on a hypersphere. We introduce a novel geometric activation function, Radial Bounding($\mathbf{x} / \max(1, \|\mathbf{x}\|_2)$), which limits the energy magnitude while preserving the phase (direction). We demonstrate mathematically that this isotropic activation maintains 1-Lipschitz continuity and prevents gradient vanishing by preserving tangential gradients. Empirically, a 100-layer Z-Plane Multi-Layer Perceptron (MLP)-entirely devoid of ReLU and LayerNorm-successfully converges on the MNIST dataset with 98.34% accuracy and absolute numerical stability, proving that bounded geometric activation alone is sufficient for stable deep learning.

Related papers

• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• Sphere Retraction Normalizations [5.536157020510715]
  We show that on the hypersphere this entire family collapses to a single scalar design choice.<n>This view places Euclidean residual connections and GeoNorm in one framework.
  arXiv  Detail & Related papers  (2026-08-02T13:23:49Z)
• Lie-Algebraic Subspace Quantization for Zero-Shot Quantum Learning and Barren-Plateau Mitigation [0.9392167468538464]
  We present a framework for zero-shot classical-to-quantum parameter transfer and manifold-based model merging.<n>We prove a Subspace Quantization Theorem that bounds the reconstruction error by geometric truncation and exact non-unitarity.<n>Experiments on IBM ibm_kobe demonstrate Hellinger fidelity of 0.987 at k-8, while subspace gradients remain resolvable up to 128 physical qubits.
  arXiv  Detail & Related papers  (2026-07-13T07:15:12Z)
• Conservation Law Breaking at the Edge of Stability: A Spectral Theory of Non-Convex Neural Network Optimization [0.0]
  We show that gradient descent reliably finds good solutions in non-dimensional network optimization.<n>We identify two regimes separated by a width-dependent dynamical transition.<n>All predictions are validated across 23 experiments.
  arXiv  Detail & Related papers  (2026-04-08T10:41:24Z)
• High-Fidelity Modeling of Stochastic Chemical Dynamics on Complex Manifolds: A Multi-Scale SIREN-PINN Framework for the Curvature-Perturbed Ginzburg-Landau Equation [0.0]
  This work establishes a new paradigm for Geometric Catalyst Design, offering a mesh-free, data-driven tool for identifying surface heterogeneity and engineering passive control strategies in turbulent chemical reactors.
  arXiv  Detail & Related papers  (2026-01-13T00:52:23Z)
• Deep Delta Learning [91.75868893250662]
  We introduce Deep Delta Learning (DDL), a novel architecture that generalizes the standard residual connection.<n>We provide a spectral analysis of this operator, demonstrating that the gate $(mathbfX)$ enables dynamic between identity mapping, projection, and geometric reflection.<n>This unification empowers the network to explicitly control the spectrum of its layer-wise transition operator, enabling the modeling of complex, non-monotonic dynamics.
  arXiv  Detail & Related papers  (2026-01-01T18:11:38Z)
• Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations [53.348574336527854]
  We investigate the neural network map as a function on the space of weights and biases.<n>We prove, for the first time, the non-singularity of the gradient descent (GD) map on the loss landscape of realistic neural network architectures.
  arXiv  Detail & Related papers  (2025-10-28T14:34:33Z)
• A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization [90.87444114491116]
  This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparametricized two-layer neural networks. We address (i) the convergence of the gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. Results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(alpha-1)$, measured in terms of the Wasserstein distance.
  arXiv  Detail & Related papers  (2024-04-18T16:46:08Z)
• Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data [66.1211659120882]
  The implicit bias towards solutions with favorable properties is believed to be a key reason why neural networks trained by gradient-based optimization can generalize well. While the implicit bias of gradient flow has been widely studied for homogeneous neural networks (including ReLU and leaky ReLU networks), the implicit bias of gradient descent is currently only understood for smooth neural networks.
  arXiv  Detail & Related papers  (2023-10-29T08:47:48Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• A Unified Algebraic Perspective on Lipschitz Neural Networks [88.14073994459586]
  This paper introduces a novel perspective unifying various types of 1-Lipschitz neural networks. We show that many existing techniques can be derived and generalized via finding analytical solutions of a common semidefinite programming (SDP) condition. Our approach, called SDP-based Lipschitz Layers (SLL), allows us to design non-trivial yet efficient generalization of convex potential layers.
  arXiv  Detail & Related papers  (2023-03-06T14:31:09Z)
• Continuous Generative Neural Networks: A Wavelet-Based Architecture in Function Spaces [1.7205106391379021]
  We study Continuous Generative Neural Networks (CGNNs) in the continuous setting. The architecture is inspired by DCGAN, with one fully connected layer, several convolutional layers and nonlinear activation functions. We present conditions on the convolutional filters and on the nonlinearity that guarantee that a CGNN is injective.
  arXiv  Detail & Related papers  (2022-05-29T11:06:29Z)
• A global convergence theory for deep ReLU implicit networks via over-parameterization [26.19122384935622]
  Implicit deep learning has received increasing attention recently. This paper analyzes the gradient flow of Rectified Linear Unit (ReLU) activated implicit neural networks.
  arXiv  Detail & Related papers  (2021-10-11T23:22:50Z)
• Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant [26.22495169129119]
  We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the Lipschitz constant. We numerically compare our scheme with standard ReLU network and its variations, PReLU and LeakyReLU.
  arXiv  Detail & Related papers  (2020-01-17T12:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.