Quotient Geometry, Effective Curvature, and Implicit Bias in Simple Shallow Neural Networks

• URL: http://arxiv.org/abs/2603.21502v1
• Date: Mon, 23 Mar 2026 02:50:30 GMT
• Title: Quotient Geometry, Effective Curvature, and Implicit Bias in Simple Shallow Neural Networks
• Abstract summary: We develop a differential-geometric framework for analyzing simple shallow networks.<n>We first characterize the symmetry and quotient structure of regular shallow-network parameters.<n>We then study gradient flows on the quotient and show that only the horizontal component of parameter motion contributes to first-order predictor evolution.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Overparameterized shallow neural networks admit substantial parameter redundancy: distinct parameter vectors may represent the same predictor due to hidden-unit permutations, rescalings, and related symmetries. As a result, geometric quantities computed directly in the ambient Euclidean parameter space can reflect artifacts of representation rather than intrinsic properties of the predictor. In this paper, we develop a differential-geometric framework for analyzing simple shallow networks through the quotient space obtained by modding out parameter symmetries on a regular set. We first characterize the symmetry and quotient structure of regular shallow-network parameters and show that the finite-sample realization map induces a natural metric on the quotient manifold. This leads to an effective notion of curvature that removes degeneracy along symmetry orbits and yields a symmetry-reduced Hessian capturing intrinsic local geometry. We then study gradient flows on the quotient and show that only the horizontal component of parameter motion contributes to first-order predictor evolution, while the vertical component corresponds purely to gauge variation. Finally, we formulate an implicit-bias viewpoint at the quotient level, arguing that meaningful complexity should be assigned to predictor classes rather than to individual parameter representatives. Our experiments confirm that ambient flatness is representation-dependent, that local dynamics are better organized by quotient-level curvature summaries, and that in underdetermined regimes, implicit bias is most naturally described in quotient coordinates.

Related papers

• Reducing Symmetry Increase in Equivariant Neural Networks [21.62431626298374]
  We provide a detailed and in-depth characterization of symmetry increase together with a principled framework for its reduction.<n>We propose practical guidelines for feature design to prevent harmful symmetry increases.<n>We provide visualizations and experiments on both synthetic datasets and the real-world QM9 dataset.
  arXiv  Detail & Related papers  (2026-08-12T12:46:25Z)
• Role-Aware Neural Convex Divergence Heads for Asymmetric Representation Learning [7.989284529490038]
  Many representation learning problems involve zero representation learning problems.<n>The proposed head is best understood as a structured and unstructured distance module for tasks where directional relations matter.<n>Overall, the proposed head is best understood as a structured and unstructured distance module for tasks where directional relations matter.
  arXiv  Detail & Related papers  (2026-07-02T06:24:18Z)
• On Higher-Order Geometric Refinements of Classical Covariance Asymptotics: An Approach via Intrinsic and Extrinsic Information Geometry [0.0]
  In curved models, including mixtures, curved exponential families, latent-variable models, and manifold-root parameter spaces, finite-sample behavior can deviate systematically from predictions.<n>We develop a coordinate-invariant, curvature-aware refinement by viewing a regular parametric family as a ((,g)) with Fisher-constrained-Rao metric.<n>We describe the resolved metric, the role of the real log canonical threshold in learning rates and posterior mean-squared error, and a curvature-based covariance expansion on the resolved space that recovers the regular theory as a special case.
  arXiv  Detail & Related papers  (2026-04-14T13:40:13Z)
• Observable Geometry of Singular Statistical Models [0.12183405753834557]
  We introduce an invariant framework based on emphobservable charts: collections of functionals of the data distribution that distinguish probability measures.<n>These charts define local coordinate systems directly on the model space, independent of parameterization.<n>Our main result establishes that, under mild regularity conditions, observable order provides a lower bound on the rate at which Kullback-Leibler divergence vanishes.
  arXiv  Detail & Related papers  (2026-04-01T15:21:52Z)
• Fubini Study geometry of representation drift in high dimensional data [0.0]
  High dimensional representation drift is commonly quantified using Euclidean or cosine distances.<n>We introduce a projective geometric view of representation drift grounded in the Fubini Study metric.<n>We show that the Fubini Study metric isolates intrinsic evolution by remaining invariant under gauge-induced fluctuations.
  arXiv  Detail & Related papers  (2026-02-01T16:00:59Z)
• Implicit bias as a Gauge correction: Theory and Inverse Design [2.9379512315137117]
  A central problem in machine learning theory is to characterize how learning dynamics select particular solutions compatible with the training objective.<n>We identify a general mechanism, in terms of an explicit correction of the learning dynamics, for the emergence of implicit biases.<n>We compute the resulting induced bias for a range of dynamics, showing how several well known results fit into a single unified framework.
  arXiv  Detail & Related papers  (2026-01-10T15:33:09Z)
• VIKING: Deep variational inference with stochastic projections [48.946143517489496]
  Variational mean field approximations tend to struggle with contemporary overparametrized deep neural networks.<n>We propose a simple variational family that considers two independent linear subspaces of the parameter space.<n>This allows us to build a fully-correlated approximate posterior reflecting the overparametrization.
  arXiv  Detail & Related papers  (2025-10-27T15:38:35Z)
• Axis-level Symmetry Detection with Group-Equivariant Representation [48.813587457507786]
  Recent heatmap-based approaches can localize potential regions of symmetry axes but often lack precision in identifying individual axes.<n>We propose a novel framework for axis-level detection of the two most common symmetry types-reflection and rotation.<n>Our method achieves state-of-the-art performance, outperforming existing approaches.
  arXiv  Detail & Related papers  (2025-08-14T15:26:53Z)
• Generalized Linear Mode Connectivity for Transformers [87.32299363530996]
  A striking phenomenon is linear mode connectivity (LMC), where independently trained models can be connected by low- or zero-loss paths.<n>Prior work has predominantly focused on neuron re-ordering through permutations, but such approaches are limited in scope.<n>We introduce a unified framework that captures four symmetry classes: permutations, semi-permutations, transformations, and general invertible maps.<n>This generalization enables, for the first time, the discovery of low- and zero-barrier linear paths between independently trained Vision Transformers and GPT-2 models.
  arXiv  Detail & Related papers  (2025-06-28T01:46:36Z)
• Relative Representations: Topological and Geometric Perspectives [50.85040046976025]
  Relative representations are an established approach to zero-shot model stitching.<n>We introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations.<n>Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes.
  arXiv  Detail & Related papers  (2024-09-17T08:09:22Z)
• The Geometry of Neural Nets' Parameter Spaces Under Reparametrization [35.5848464226014]
  We study the invariance of neural nets under reparametrization from the perspective of Riemannian geometry. We discuss implications for measuring the flatness of minima, optimization, and for probability-density.
  arXiv  Detail & Related papers  (2023-02-14T22:48:24Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• Semiparametric Nonlinear Bipartite Graph Representation Learning with Provable Guarantees [106.91654068632882]
  We consider the bipartite graph and formalize its representation learning problem as a statistical estimation problem of parameters in a semiparametric exponential family distribution. We show that the proposed objective is strongly convex in a neighborhood around the ground truth, so that a gradient descent-based method achieves linear convergence rate. Our estimator is robust to any model misspecification within the exponential family, which is validated in extensive experiments.
  arXiv  Detail & Related papers  (2020-03-02T16:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.