Scale redundancy and soft gauge fixing in positively homogeneous neural networks

• URL: http://arxiv.org/abs/2602.14729v1
• Date: Mon, 16 Feb 2026 13:21:49 GMT
• Title: Scale redundancy and soft gauge fixing in positively homogeneous neural networks
• Authors: Rodrigo Carmo Terin,
• Abstract summary: Neural networks with positively homogeneous activations exhibit an exact continuous reparametrization symmetry.<n>We introduce gauge-adapted coordinates that separate invariant and scale-imbalance directions.<n>Inspired by gauge fixing in field theory, we introduce a soft orbit-selection functional acting only on redundant scale coordinates.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks with positively homogeneous activations exhibit an exact continuous reparametrization symmetry: neuron-wise rescalings generate parameter-space orbits along which the input--output function is invariant. We interpret this symmetry as a gauge redundancy and introduce gauge-adapted coordinates that separate invariant and scale-imbalance directions. Inspired by gauge fixing in field theory, we introduce a soft orbit-selection (norm-balancing) functional acting only on redundant scale coordinates. We show analytically that it induces dissipative relaxation of imbalance modes to preserve the realized function. In controlled experiments, this orbit-selection penalty expands the stable learning-rate regime and suppresses scale drift without changing expressivity. These results establish a structural link between gauge-orbit geometry and optimization conditioning, providing a concrete connection between gauge-theoretic concepts and machine learning.

Related papers

• Toward Manifest Relationality in Transformers via Symmetry Reduction [0.0]
  Transformer models contain substantial internal redundancy.<n>Recent approaches address this by explicitly breaking symmetry.<n>We propose a complementary framework based on symmetry reduction.
  arXiv  Detail & Related papers  (2026-02-21T19:43:17Z)
• Steering Dynamical Regimes of Diffusion Models by Breaking Detailed Balance [2.336538451986937]
  We show that deliberately breaking detailed balance in generative diffusion processes can accelerate the reverse process without changing the stationary distribution.<n>We analyze how such non-reversible control reshapes the macroscopic dynamical regimes of the phase transitions recently identified in generative diffusion models.
  arXiv  Detail & Related papers  (2026-02-17T03:05:37Z)
• Recurrent Equivariant Constraint Modulation: Learning Per-Layer Symmetry Relaxation from Data [36.287199718605]
  Equivariant neural networks exploit underlying task symmetries to improve generalization.<n>We propose Recurrent Equivariant Constraint Modulation (RECM), a layer-wise constraint modulation mechanism.<n>RECM learns appropriate relaxation levels solely from the training signal and the symmetry properties of each layer's input-target distribution.
  arXiv  Detail & Related papers  (2026-02-02T21:59:35Z)
• VIKING: Deep variational inference with stochastic projections [48.946143517489496]
  Variational mean field approximations tend to struggle with contemporary overparametrized deep neural networks.<n>We propose a simple variational family that considers two independent linear subspaces of the parameter space.<n>This allows us to build a fully-correlated approximate posterior reflecting the overparametrization.
  arXiv  Detail & Related papers  (2025-10-27T15:38:35Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• A Regularization-Guided Equivariant Approach for Image Restoration [46.44312175792672]
  Equivariant and invariant deep learning models have been developed to exploit intrinsic symmetries in data.<n>These methods often suffer from limited representation accuracy and rely on strict symmetry assumptions that may not hold in practice.<n>We propose a rotation-equivariant regularization strategy that adaptively enforces the appropriate symmetry constraints on the data while preserving the network's representational accuracy.
  arXiv  Detail & Related papers  (2025-05-26T10:30:26Z)
• Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy [75.15685966213832]
  We analyze the rich directional structure of optimization trajectories represented by their pointwise parameters. We show that training only scalar batchnorm parameters some while into training matches the performance of training the entire network.
  arXiv  Detail & Related papers  (2024-03-12T07:32:47Z)
• Symmetry Breaking and Equivariant Neural Networks [17.740760773905986]
  We introduce a novel notion of'relaxed equiinjection' We show how to incorporate this relaxation into equivariant multilayer perceptronrons (E-MLPs) The relevance of symmetry breaking is then discussed in various application domains.
  arXiv  Detail & Related papers  (2023-12-14T15:06:48Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Message-Passing Neural Quantum States for the Homogeneous Electron Gas [41.94295877935867]
  We introduce a message-passing-neural-network-based wave function Ansatz to simulate extended, strongly interacting fermions in continuous space. We demonstrate its accuracy by simulating the ground state of the homogeneous electron gas in three spatial dimensions.
  arXiv  Detail & Related papers  (2023-05-12T04:12:04Z)
• Model Reduction for Nonlinear Systems by Balanced Truncation of State and Gradient Covariance [0.0]
  We find low-dimensional systems of coordinates for model reduction that balance adjoint-based information about the system's sensitivity with the variance of states along trajectories. We demonstrate these techniques on a simple, yet challenging three-dimensional system and a nonlinear axisymmetric jet flow simulation with $105$ state variables.
  arXiv  Detail & Related papers  (2022-07-28T21:45:08Z)
• Optimizing Mode Connectivity via Neuron Alignment [84.26606622400423]
  Empirically, the local minima of loss functions can be connected by a learned curve in model space along which the loss remains nearly constant. We propose a more general framework to investigate effect of symmetry on landscape connectivity by accounting for the weight permutations of networks being connected.
  arXiv  Detail & Related papers  (2020-09-05T02:25:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.