The Geometric Cost of Normalization: Affine Bounds on the Bayesian Complexity of Neural Networks

• URL: http://arxiv.org/abs/2603.27432v1
• Date: Sat, 28 Mar 2026 22:15:45 GMT
• Title: The Geometric Cost of Normalization: Affine Bounds on the Bayesian Complexity of Neural Networks
• Abstract summary: We prove that LayerNorm's mean-centering step, by confining data to a linear hyperplane (through the origin), reduces the Local Learning Coefficient (LLC) of the subsequent weight matrix by exactly $m/2$.<n>We further show that Softmax simplex data introduces a "smuggled bias" that activates the same $m/2$ LLC drop when paired with an explicit downstream bias.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LayerNorm and RMSNorm impose fundamentally different geometric constraints on their outputs - and this difference has a precise, quantifiable consequence for model complexity. We prove that LayerNorm's mean-centering step, by confining data to a linear hyperplane (through the origin), reduces the Local Learning Coefficient (LLC) of the subsequent weight matrix by exactly $m/2$ (where $m$ is its output dimension); RMSNorm's projection onto a sphere preserves the LLC entirely. This reduction is structurally guaranteed before any training begins, determined by data manifold geometry alone. The underlying condition is a geometric threshold: for the codimension-one manifolds we study, the LLC drop is binary -- any non-zero curvature, regardless of sign or magnitude, is sufficient to preserve the LLC, while only affinely flat manifolds cause the drop. At finite sample sizes this threshold acquires a smooth crossover whose width depends on how much of the data distribution actually experiences the curvature, not merely on whether curvature exists somewhere. We verify both predictions experimentally with controlled single-layer scaling experiments using the wrLLC framework. We further show that Softmax simplex data introduces a "smuggled bias" that activates the same $m/2$ LLC drop when paired with an explicit downstream bias, proved via the affine symmetry extension of the main theorem and confirmed empirically.

Related papers

• Particle Dynamics of Flow Matching and Classifier-Free Guidance from a Stagewise Geometry Perspective [3.9423546337949253]
  Flow matching, together with guidance (CFG), is widely used in generative modeling.<n>We establish a unified stagewise geometric theory attraction attraction for both dynamics.<n>We further show that a general time schedule $a(t)$ replaces the $O(1-t)$ decay by $O(1-a(t)$.
  arXiv  Detail & Related papers  (2026-09-07T02:33:36Z)
• Statistically Meaningful Geometry (SMG) Beyond the Euclidean Paradigm, with Application to Generative AI [0.7766379063545538]
  We introduce the Statistically Meaningful Geometry (SMG) framework, an information-geometric paradigm lifting parametric models into infinite-dimensional minimization non-asymptotic statistics.<n>We prove that under connection-filtered pre-training, out-of-distribution predictive variance is strictly upper-bounded by the finite diameter of the identifiable base manifold $mathcalB$, establishing a hard geometric containment of generative hallucinations.
  arXiv  Detail & Related papers  (2026-07-03T13:46:31Z)
• Intrinsic Wasserstein Rates for Score-Based Generative Models on Smooth Manifolds [61.14405512940818]
  We show that Score-based generative models are trained in high-dimensional ambient spaces.<n>We use a ReLU implementation of nearest-projection coordinates via finite intrinsic anchors and Gauss Newton.
  arXiv  Detail & Related papers  (2026-05-15T10:20:05Z)
• A Mean Curvature Approach to Boundary Detection: Geometric Insights for Unsupervised Learning [52.452902154360565]
  We introduce Mean Curvature Boundary Points (MCBP), a novel geometric framework grounded in Geometric Machine Learning.<n>MCBP explicitly models the intrinsic curvature of the data manifold and computes pointwise mean curvature without requiring principled manifold parametrization.<n>Experiments on synthetic and real-world datasets demonstrate that MCBP consistently improves clustering performance.
  arXiv  Detail & Related papers  (2026-05-05T20:19:09Z)
• Information-Geometric Decomposition of Generalization Error in Unsupervised Learning [0.0]
  We decompose the Kullback--Leibler generalization error (GE) of unsupervised learning into three non-negative components: model error, data bias, and variance.<n>The decomposition is exact for any e-flat model class.
  arXiv  Detail & Related papers  (2026-04-14T06:23:18Z)
• All ERMs Can Fail in Stochastic Convex Optimization Lower Bounds in Linear Dimension [14.982451024975733]
  We show that there exists an instance in which the sample size is linear in the dimension, learning is possible, but the Empirical Risk Minimizer is likely to be unique and to overfit.<n>We provide a novel generalization lower bound of $left(sqrtT/m1.5right)$ for Gradient Descent, where $$ is the learning rate, $T$ is the horizon and $m$ is the sample size.
  arXiv  Detail & Related papers  (2026-02-09T07:33:01Z)
• An approach to Fisher-Rao metric for infinite dimensional non-parametric information geometry [0.6138671548064355]
  Being infinite dimensional, non-parametric information geometry has long faced an "intractability barrier"<n>This paper introduces a novel framework to resolve the intractability with an Orthogonal Decomposition of the Tangent Space.<n>By defining the Information Capture Ratio, we provide a rigorous method for estimating intrinsic dimensionality in high-dimensional data.
  arXiv  Detail & Related papers  (2025-12-25T00:18:41Z)
• FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivative [0.0]
  textttFedMPDD (textbfFederated Learning via textbfMulti-textbfProjected textbfDirectional textbfDerivatives)<n>textttFedMPDD encodes each client's high-dimensional gradient by computing its directional derivatives along multiple random vectors.
  arXiv  Detail & Related papers  (2025-12-23T22:25:11Z)
• Phase-space entropy at acquisition reflects downstream learnability [54.4100065023873]
  We propose an acquisition-level scalar $S_mathcal B$ based on instrument-resolved phase space.<n>We show theoretically that (S_mathcal B) correctly identifies the phase-space coherence of periodic sampling.<n>$|S_mathcal B|$ consistently ranks sampling geometries and predicts downstream reconstruction/recognition difficulty emphwithout training.
  arXiv  Detail & Related papers  (2025-12-22T10:03:51Z)
• Non-convex matrix sensing: Breaking the quadratic rank barrier in the sample complexity [11.412228884390784]
  We show that factorized gradient descent scales to the truth at the number of samples.<n>We extend our theory to the noisy setting, where we show that with noisy measurements the gradient descents are only weakly dependent on the measurement matrices.
  arXiv  Detail & Related papers  (2024-08-20T14:09:28Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Continuum Limits of Ollivier's Ricci Curvature on data clouds: pointwise consistency and global lower bounds [1.1126342180866644]
  We study the relationship between the curvature of a random geometric graph built from $X$ and the curvature of the manifold $M$ via continuum limits of Ollivier's discrete Ricci curvature. We discuss applications of the global discrete curvature bounds to contraction properties of heat kernels on graphs, as well as implications for manifold learning from data clouds.
  arXiv  Detail & Related papers  (2023-07-05T15:45:53Z)
• Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation [52.73824786627612]
  This paper establishes new convergence results for textitgeodesic strongly monotone games.<n>Our key result shows that RGD attains last-iterate linear convergence in a textitgeometry-agnostic fashion.<n>Overall, this paper presents the first geometry-agnostic last-iterate convergence analysis for games beyond the Euclidean settings.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• Effective Minkowski Dimension of Deep Nonparametric Regression: Function Approximation and Statistical Theories [70.90012822736988]
  Existing theories on deep nonparametric regression have shown that when the input data lie on a low-dimensional manifold, deep neural networks can adapt to intrinsic data structures. This paper introduces a relaxed assumption that input data are concentrated around a subset of $mathbbRd$ denoted by $mathcalS$, and the intrinsic dimension $mathcalS$ can be characterized by a new complexity notation -- effective Minkowski dimension.
  arXiv  Detail & Related papers  (2023-06-26T17:13:31Z)
• Restricted Strong Convexity of Deep Learning Models with Smooth Activations [31.003601717265006]
  We study the problem of optimization of deep learning models with smooth activation functions. We introduce a new analysis of optimization based on Restricted Strong Convexity (RSC) Ours is the first result on establishing geometric convergence of GD based on RSC for deep learning models.
  arXiv  Detail & Related papers  (2022-09-29T21:24:26Z)
• The Generalized Lasso with Nonlinear Observations and Generative Priors [63.541900026673055]
  We make the assumption of sub-Gaussian measurements, which is satisfied by a wide range of measurement models. We show that our result can be extended to the uniform recovery guarantee under the assumption of a so-called local embedding property.
  arXiv  Detail & Related papers  (2020-06-22T16:43:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.