Hidden Gauge Controls Feature Specialization in ReLU Networks

• URL: http://arxiv.org/abs/2608.06766v1
• Date: Fri, 07 Aug 2026 03:39:09 GMT
• Title: Hidden Gauge Controls Feature Specialization in ReLU Networks
• Abstract summary: Training changes a network's predictions while allocating task-relevant structure across its internal units.<n>We call the identity of that neuron feature ownership and ask whether it can be controlled by a parameter choice invisible to the initial predictor.<n>We prove global selection and functional pruning, extend finite-time selection to visible perturbations and small-step full-batch gradient descent.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training changes a network's predictions while allocating task-relevant structure across its internal units. In an overparameterized ReLU network, several neurons can begin with exactly the same functional role, yet one may acquire a teacher feature while the others become redundant. We call the identity of that neuron feature ownership and ask whether it can be controlled by a parameter choice invisible to the initial predictor. In a tractable Gaussian teacher--student model, we fix the complete initial function and vary only a positive-homogeneous scaling gauge. Opposite gauges produce distinct feature trajectories and a sharp $Θ(D^2)$ separation in specialization time that no global change of clock can explain. Among any fixed number of initially duplicate students, assigning the favorable gauge to one neuron deterministically selects it as the owner and drives the remaining functional contribution to zero. An exact reaction--transport decomposition attributes the effect to different mobilities for changing a feature's coefficient and direction. We prove global selection and functional pruning, extend finite-time selection to visible perturbations and small-step full-batch gradient descent, and verify the predicted loss, alignment, pruning, and dissipation trajectories in population and finite-sample training. The initial predictor therefore determines neither when the feature is learned nor which neuron learns it.

Related papers

• Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy [37.56003520528009]
  In teacher-student networks, ME recovers the minimal teacher-consistent representation and retains extra units.<n>In a non-linear Gaussian process task, ME selects coherent functional-class mappings whose preferred class shifts across training.<n>On this task and on translation-augmented MNIST, ME-selectedworks outperform random subsets of equal size.
  arXiv  Detail & Related papers  (2026-09-04T13:25:58Z)
• Debiased neural operators for estimating functionals [55.79111577184338]
  We introduce DOPE (debiased neural operator): a semivalued estimator for target quantities of solution trajectories obtained from neural operators.<n>We show that, in contrast to DOPE, naive plug-in estimation can suffer from first-order bias.<n>For this, we derive a novel one-step, Neyman-orthogonal estimator that treats the neural operator as a high-dimensional nuisance mapping between function spaces.
  arXiv  Detail & Related papers  (2026-04-21T10:02:36Z)
• Exploring the Dimensions of a Variational Neuron [0.0]
  We introduce EVE, a variational distributional neuron formulated as a local probabilistic computational unit.<n>Eve is instrumented with internal diagnostics and constraints, including effective KL, a target band on mu2, out-of-band fractions, and indicators of drift and collapse.<n>Overall, the paper provides an experimentally grounded first map of the design space opened by a variational neuron.
  arXiv  Detail & Related papers  (2026-03-14T09:09:13Z)
• A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities [30.737171081270322]
  We study how fully-connected two-layer neural networks adapt to the target function after a single, but aggressive, gradient descent step. This provides a sharp description of the impact of feature learning in the generalization of two-layer neural networks, beyond the random features and lazy training regimes.
  arXiv  Detail & Related papers  (2024-10-24T17:24:34Z)
• Dimension-independent learning rates for high-dimensional classification problems [53.622581586464634]
  We show that every $RBV2$ function can be approximated by a neural network with bounded weights. We then prove the existence of a neural network with bounded weights approximating a classification function.
  arXiv  Detail & Related papers  (2024-09-26T16:02:13Z)
• Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs [5.7166378791349315]
  We prove that, for the fundamental regression task of learning a single neuron, training a one-hidden layer ReLU network converges to zero loss. We also show and characterise a surprising distinction in this setting between interpolator networks of minimal rank and those of minimal Euclidean norm.
  arXiv  Detail & Related papers  (2023-06-10T16:36:22Z)
• Modeling Implicit Bias with Fuzzy Cognitive Maps [0.0]
  This paper presents a Fuzzy Cognitive Map model to quantify implicit bias in structured datasets. We introduce a new reasoning mechanism equipped with a normalization-like transfer function that prevents neurons from saturating.
  arXiv  Detail & Related papers  (2021-12-23T17:04:12Z)
• The Causal Neural Connection: Expressiveness, Learnability, and Inference [125.57815987218756]
  An object called structural causal model (SCM) represents a collection of mechanisms and sources of random variation of the system under investigation. In this paper, we show that the causal hierarchy theorem (Thm. 1, Bareinboim et al., 2020) still holds for neural models. We introduce a special type of SCM called a neural causal model (NCM), and formalize a new type of inductive bias to encode structural constraints necessary for performing causal inferences.
  arXiv  Detail & Related papers  (2021-07-02T01:55:18Z)
• Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approach [84.29777236590674]
  We study the estimation of causal parameters when not all confounders are observed and instead negative controls are available. Recent work has shown how these can enable identification and efficient estimation via two so-called bridge functions.
  arXiv  Detail & Related papers  (2021-03-25T17:59:19Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• No one-hidden-layer neural network can represent multivariable functions [0.0]
  In a function approximation with a neural network, an input dataset is mapped to an output index by optimizing the parameters of each hidden-layer unit. We present constraints on the parameters and its second derivative by constructing a continuum version of a one-hidden-layer neural network with the rectified linear unit (ReLU) activation function.
  arXiv  Detail & Related papers  (2020-06-19T06:46:54Z)
• Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory [110.99247009159726]
  Temporal-difference and Q-learning play a key role in deep reinforcement learning, where they are empowered by expressive nonlinear function approximators such as neural networks. In particular, temporal-difference learning converges when the function approximator is linear in a feature representation, which is fixed throughout learning, and possibly diverges otherwise.
  arXiv  Detail & Related papers  (2020-06-08T17:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.