Most ReLU Networks Admit Identifiable Parameters

• URL: http://arxiv.org/abs/2605.03601v1
• Date: Tue, 05 May 2026 10:25:28 GMT
• Title: Most ReLU Networks Admit Identifiable Parameters
• Abstract summary: We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation.<n>Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters.
• Score: 14.626408075188005
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation. To analyze hidden redundancies beyond these standard symmetries, we introduce a framework based on weighted polyhedral complexes. Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters. This implies that the functional dimension of every such architecture is exactly the number of parameters minus the number of hidden neurons. We further show that minimal functional representations can still have non-trivial parameter redundancies. Finally, we establish a generic depth hierarchy, whereby for an open set of parameters the realized function cannot be represented generically by any shallower network.

Related papers

• Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity [19.53343888629072]
  We develop a unified function space theory of deep fully connected neural networks.<n>Our framework provides a meaningful notion of functional complexity for deep networks.<n>We show that deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction.
  arXiv  Detail & Related papers  (2026-07-06T18:29:35Z)
• A Complete Symmetry Classification of Shallow ReLU Networks [0.0]
  We study the study of distinct parameters in neural architectures which realize the same function.<n>We exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.
  arXiv  Detail & Related papers  (2026-04-15T16:18:44Z)
• FlexPara: Flexible Neural Surface Parameterization [71.65203972602673]
  This paper introduces FlexPara, an unsupervised neural optimization framework to achieve both global and multi-chart surface parameterizations.<n>We ingeniously design and combine a series of geometrically-interpretable sub-networks, with specific functionalities, to construct a bi-directional cycle mapping framework for global parameterization.<n>Experiments demonstrate the universality, superiority, and inspiring potential of our neural surface parameterization paradigm.
  arXiv  Detail & Related papers  (2025-04-27T12:30:08Z)
• Geometry-induced Implicit Regularization in Deep ReLU Neural Networks [0.0]
  Implicit regularization phenomena, which are still not well understood, occur during optimization. We study the geometry of the output set as parameters vary. We prove that the batch functional dimension is almost surely determined by the activation patterns in the hidden layers.
  arXiv  Detail & Related papers  (2024-02-13T07:49:57Z)
• Hidden symmetries of ReLU networks [17.332539115959708]
  In some networks, the only symmetries are permutation of neurons in a layer and positive scaling of parameters at a neuron, while other networks admit additional hidden symmetries. In this work, we prove that, for any network architecture where no layer is narrower than the input, there exist parameter settings with no hidden symmetries.
  arXiv  Detail & Related papers  (2023-06-09T18:07:06Z)
• Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks [0.0]
  A neural network parameter's functional equivalence class is the set of parameters implementing the same input--output function. We give an algorithmic characterisation of unit redundancies and reducible functional equivalence classes for a single-hidden-layer hyperbolic tangent architecture.
  arXiv  Detail & Related papers  (2023-05-08T23:29:12Z)
• Expand-and-Cluster: Parameter Recovery of Neural Networks [9.497862562614666]
  We show that the incoming weight vector of each neuron is identifiable up to sign or scaling, depending on the activation function. Our novel method 'Expand-and-Cluster' can identify weights of a target network for all commonly used activation functions.
  arXiv  Detail & Related papers  (2023-04-25T13:14:20Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• RepMode: Learning to Re-parameterize Diverse Experts for Subcellular Structure Prediction [54.69195221765405]
  In biological research, fluorescence staining is a key technique to reveal the locations and morphology of subcellular structures. In this paper, we model it as a deep learning task termed subcellular structure prediction (SSP), aiming to predict the 3D fluorescent images of multiple subcellular structures from a 3D transmitted-light image. We propose RepMode, a network that dynamically organizes its parameters with task-aware priors to handle specified single-label prediction tasks.
  arXiv  Detail & Related papers  (2022-12-20T08:17:08Z)
• A Functional Perspective on Learning Symmetric Functions with Neural Networks [48.80300074254758]
  We study the learning and representation of neural networks defined on measures. We establish approximation and generalization bounds under different choices of regularization. The resulting models can be learned efficiently and enjoy generalization guarantees that extend across input sizes.
  arXiv  Detail & Related papers  (2020-08-16T16:34:33Z)
• Neural Parameter Allocation Search [57.190693718951316]
  Training neural networks requires increasing amounts of memory. Existing methods assume networks have many identical layers and utilize hand-crafted sharing strategies that fail to generalize. We introduce Neural Allocation Search (NPAS), a novel task where the goal is to train a neural network given an arbitrary, fixed parameter budget. NPAS covers both low-budget regimes, which produce compact networks, as well as a novel high-budget regime, where additional capacity can be added to boost performance without increasing inference FLOPs.
  arXiv  Detail & Related papers  (2020-06-18T15:01:00Z)
• When Residual Learning Meets Dense Aggregation: Rethinking the Aggregation of Deep Neural Networks [57.0502745301132]
  We propose Micro-Dense Nets, a novel architecture with global residual learning and local micro-dense aggregations. Our micro-dense block can be integrated with neural architecture search based models to boost their performance.
  arXiv  Detail & Related papers  (2020-04-19T08:34:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.