Sphere Retraction Normalizations

• URL: http://arxiv.org/abs/2608.02668v1
• Date: Sun, 02 Aug 2026 13:23:49 GMT
• Title: Sphere Retraction Normalizations
• Abstract summary: We show that on the hypersphere this entire family collapses to a single scalar design choice.<n>This view places Euclidean residual connections and GeoNorm in one framework.
• Score: 5.536157020510715
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Residual connections are the de facto mechanism for training deep neural networks stably. Geodesic Normalization (GeoNorm) recasts them on a Riemannian manifold, orthogonalizing each layer output against the current hidden state and applying the resulting update through the Riemannian exponential map. Every hidden state thus keeps a constant $\ell_{2}$-norm, confining the residual stream to a hypersphere. The exponential map, however, is only one member of a broad family of retraction maps. We show that on the hypersphere this entire family collapses to a single scalar design choice. What distinguishes one retraction from another is only how the magnitude of an update is converted into a rotation angle within the plane spanned by the hidden state and the update. This view places Euclidean residual connections and GeoNorm in one framework. Instantiating it with the metric projection retraction and the Cayley retraction yields Proj-SpheretNorm and Cay-SpheretNorm, which are exactly norm-preserving yet require only algebraic operations. Both prove to be members of a one-parameter family of angular retractions, $p$-SpheretNorm, whose rotation angle saturates rather than growing without bound. The two methods above are recovered exactly at $p = 1$ and $p = 2$, while the identity map and GeoNorm arise only as limits at either end. On nanoGPT, all three methods outperform existing lightweight deep connection schemes, and the best validation loss is attained at finite $p$, indicating that the exponential map is not the preferred retraction for spherical residual streams but merely one end of a spectrum.

Related papers

• The Capacity of Entanglement and Holographic Entropies at Finite Resources [0.0]
  Ryu-Takayanagi formula equates the area of a minimal surface with the von Neumann entropy of a boundary subregion.<n>Fannes-Audenaert answers with a Hilbert-space dimension, which diverges as the cutoff is removed however close the two states are.<n>We replace it with the capacity of entanglement, the variance of the modular energy, whose square root grows like the square root of the entangling area.
  arXiv  Detail & Related papers  (2026-08-26T20:02:55Z)
• Quantum Markov Chains for an Asymmetric Mixed Ising-XY Model on a Cayley Tree [0.0]
  We study a mixed quantum Ising-$XY$ model on the semi-infinite rooted Cayley tree of order two.<n>We derive the translation-invariant boundary equation and compute the associated local transfer operator.<n>We show that the reduced boundary-law dynamics has no admissible periodic points of period greater than one.
  arXiv  Detail & Related papers  (2026-07-15T20:15:08Z)
• Optimization Geometrodynamics: Variational Reduction and Interaction Curvature [0.0]
  We develop geometrodynamics as a variational theory of this hidden determinant pushforward.<n>For affine pre-reductions, the induced interaction curvature is the negative-semidefinite operator $-G*H-1G$.<n>We prove a global analytic bundle with closed totally geodesic fibers and a unique analytic nearest-controller section.
  arXiv  Detail & Related papers  (2026-07-07T18:42:00Z)
• SILAGE: Memory-Efficient, Full-Gradient-Free Nonconvex Optimization for Nested Finite Sums [51.49970814177172]
  Empirical risk on datasets naturally exhibits similarities in $N=nm$ total samples.<n>We provide an analysis that avoids pessimistic convergence analysis.<n>Our results improve existing state-of-the-art regimes.
  arXiv  Detail & Related papers  (2026-06-14T14:11:07Z)
• Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm [3.0459410574367314]
  Modern deep neural networks rely on Euclidean scalar activations (e.g., ReLU) to prevent gradient instability in deep architectures.<n>We propose the Z-Plane Neural Network, which maps hidden states into 2D phasor bundles on a hypersphere.<n>We demonstrate mathematically that this isotropic activation maintains 1-Lipschitz continuity and prevents gradient vanishing by preserving tangential gradients.
  arXiv  Detail & Related papers  (2026-06-14T08:22:28Z)
• Towards Scalable Persistence-Based Topological Optimization [44.16669776030478]
  Persistence-based topological optimization deforms a point cloud $X subset mathbbRd$ by minimizing objectives of the form $L(X) = ell(mathrmDgm(X))$, where $mathrmDgm(X)$ is a persistence diagram.<n>In practice, optimization is limited by two coupled issues: persistent homology is typically computed on subsamples, and the resulting topological gradients are highly sparse, with only a few anchor points receiving nonzero updates.
  arXiv  Detail & Related papers  (2026-05-09T15:47:20Z)
• Nodal algebraic curves and entropy diagnostics in degenerate two-dimensional harmonic-oscillator shells [0.0]
  We show that degenerate quantum eigenspaces can support substantial changes in nodal geometry at fixed energy.<n>We combine these criteria with entropy diagnostics to identify strata where topology-changing events can occur.<n>The framework suggests experimentally reconstructible signatures for real-phase Hermite--Gaussian structured light and approximately isotropic trapped motional systems.
  arXiv  Detail & Related papers  (2026-04-30T17:15:26Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward [66.81579829897392]
  We propose a novel offline reinforcement learning algorithm called Pessimistic vAlue iteRaTion with rEward Decomposition (PARTED) PARTED decomposes the trajectory return into per-step proxy rewards via least-squares-based reward redistribution, and then performs pessimistic value based on the learned proxy reward. To the best of our knowledge, PARTED is the first offline RL algorithm that is provably efficient in general MDP with trajectory-wise reward.
  arXiv  Detail & Related papers  (2022-06-13T19:11:22Z)
• Point2Mesh: A Self-Prior for Deformable Meshes [83.31236364265403]
  We introduce Point2Mesh, a technique for reconstructing a surface mesh from an input point cloud. The self-prior encapsulates reoccurring geometric repetitions from a single shape within the weights of a deep neural network. We show that Point2Mesh converges to a desirable solution; compared to a prescribed smoothness prior, which often becomes trapped in undesirable local minima.
  arXiv  Detail & Related papers  (2020-05-22T10:01:04Z)
• Anisotropy-mediated reentrant localization [62.997667081978825]
  We consider a 2d dipolar system, $d=2$, with the generalized dipole-dipole interaction $sim r-a$, and the power $a$ controlled experimentally in trapped-ion or Rydberg-atom systems. We show that the spatially homogeneous tilt $beta$ of the dipoles giving rise to the anisotropic dipole exchange leads to the non-trivial reentrant localization beyond the locator expansion.
  arXiv  Detail & Related papers  (2020-01-31T19:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.