Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization

• URL: http://arxiv.org/abs/2607.18930v1
• Date: Tue, 21 Jul 2026 10:12:03 GMT
• Title: Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization
• Abstract summary: We analyze functional equivalence and geometric diversity of neural network approximations to a few elementary mathematical functions.<n>The study reveals large equivalence classes of functionally indistinguishable yet geometrically diverse networks.<n>A model select criterion is proposed for identifying optimal models based on parsimony, ease of estimation, and inference efficiency.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error. However, the uniqueness of such neural network representations is not guaranteed, raising questions about practical identifiability. In this work, we address this concern by analyzing functional equivalence and geometric diversity of neural network approximations to a few elementary mathematical functions. The analysis includes an extensive study of single-layer neural networks and multilayer perceptrons under noisy and noise-free conditions. Beyond just network capacity, we study the geometric properties through the lens of sloppiness, characterized by the eigen spectrum of the Hessian of the cost function and the effective rank to quantify the dimensionality of parameter space. The study reveals large equivalence classes of functionally indistinguishable yet geometrically diverse networks that consistently exhibit low effective rank and structural redundancy. Finally, a model select criterion is proposed for identifying optimal models based on parsimony, ease of estimation, and inference efficiency.

Related papers

• Approximation Theory for Neural Networks: Old and New [1.6300219183960216]
  Universal approximations provide a mathematical explanation for the expressive power of neural networks.<n>We review classical density results for single-hidden-layer networks, as well as quantitative bounds that relate approximation error to network size.<n>Recent developments on Kolmogorov--Arnold Networks (KANs) offer an alternative architectural paradigm.
  arXiv  Detail & Related papers  (2026-05-20T17:42:34Z)
• GRIFDIR: Graph Resolution-Invariant FEM Diffusion Models in Function Spaces over Irregular Domains [56.121725064621295]
  Score-based diffusion models in function spaces offer a principled framework for modelling function-valued data.<n>However, practical implementations have struggled to fully realise these benefits.<n>We propose a novel architecture that represents generalised convolutional kernels as finite graph kernels.
  arXiv  Detail & Related papers  (2026-05-05T08:33:52Z)
• Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension [27.6052958612001]
  We propose a framework that employs convolutional architectures to extract sparse features from a finite number of samples.<n>We show that sparse approximators enable stable recovery from discrete samples.<n>They also provide new theoretical insights into how sparsity can alleviate the curse of dimensionality in functional learning.
  arXiv  Detail & Related papers  (2026-04-08T07:42:02Z)
• Explainable Neural Networks with Guarantees: A Sparse Estimation Approach [11.142723510517778]
  This paper introduces a novel approach to constructing an explainable neural network that harmonizes predictiveness and explainability.<n>Our model, termed SparXnet, is designed as a linear combination of a sparse set of jointly learned features.<n>Our research paves the way for further research on sparse and explainable neural networks with guarantee.
  arXiv  Detail & Related papers  (2025-01-02T12:10:17Z)
• Deep Neural Networks are Adaptive to Function Regularity and Data Distribution in Approximation and Estimation [8.284464143581546]
  We study how deep neural networks can adapt to different regularity in functions across different locations and scales. Our results show that deep neural networks are adaptive to different regularity of functions and nonuniform data distributions.
  arXiv  Detail & Related papers  (2024-06-08T02:01:50Z)
• Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory [64.06519549649495]
  We provide the definition of what we call functionally equivalent features. These features produce equivalent output under certain transformations. We propose an efficient algorithm named Iterative Feature Merging.
  arXiv  Detail & Related papers  (2023-10-10T16:27:12Z)
• Multilayer Multiset Neuronal Networks -- MMNNs [55.2480439325792]
  The present work describes multilayer multiset neuronal networks incorporating two or more layers of coincidence similarity neurons. The work also explores the utilization of counter-prototype points, which are assigned to the image regions to be avoided.
  arXiv  Detail & Related papers  (2023-08-28T12:55:13Z)
• Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems [3.660098145214465]
  We develop an approximation theory for compositional functions and their neural network approximations. We identify a set of key features of compositional functions and the relationship between the features and the complexity of neural networks. In addition to function approximations, we prove several formulae of error upper bounds for neural networks.
  arXiv  Detail & Related papers  (2020-12-03T04:40:25Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory [110.99247009159726]
  Temporal-difference and Q-learning play a key role in deep reinforcement learning, where they are empowered by expressive nonlinear function approximators such as neural networks. In particular, temporal-difference learning converges when the function approximator is linear in a feature representation, which is fixed throughout learning, and possibly diverges otherwise.
  arXiv  Detail & Related papers  (2020-06-08T17:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.