Approximation Theory for Neural Networks: Old and New

• URL: http://arxiv.org/abs/2605.21451v1
• Date: Wed, 20 May 2026 17:42:34 GMT
• Title: Approximation Theory for Neural Networks: Old and New
• Abstract summary: Universal approximations provide a mathematical explanation for the expressive power of neural networks.<n>We review classical density results for single-hidden-layer networks, as well as quantitative bounds that relate approximation error to network size.<n>Recent developments on Kolmogorov--Arnold Networks (KANs) offer an alternative architectural paradigm.
• Score: 1.6300219183960216
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Universal approximation theorems provide a mathematical explanation for the expressive power of neural networks. They assert that, under mild conditions on the activation function, feedforward neural networks are dense in broad function classes, such as continuous functions on compact subsets of $\mathbb{R}^d$, $L^p$ spaces, or Sobolev spaces. Over the past four decades, these qualitative universality results have evolved into a rich quantitative theory addressing approximation rates, parameter efficiency, and the role of architectural features such as depth and width. This survey presents several glimpses into this theory. We review classical density results for single-hidden-layer networks, as well as quantitative bounds that relate approximation error to network size and smoothness assumptions on target functions. Particular emphasis is placed on depth--width trade-offs and on results demonstrating that deeper architectures can achieve superior parameter efficiency for structured function classes. In addition to standard feedforward neural networks, we also review recent developments on Kolmogorov--Arnold Networks (KANs), which offer an alternative architectural paradigm and whose approximation-theoretic properties have begun to attract significant theoretical attention.

Related papers

• Dense Neural Networks are not Universal Approximators [53.27010448621372]
  We show that dense neural networks do not possess universality of arbitrary continuous functions.<n>We consider ReLU neural networks subject to natural constraints on weights and input and output dimensions.
  arXiv  Detail & Related papers  (2026-02-07T16:52:38Z)
• Distributionally robust approximation property of neural networks [0.0]
  We prove that neural networks are dense in Orlicz spaces, thereby extending classical universal approximation theorems even beyond the traditional $Lp$-setting.<n>The covered classes of neural networks include widely used architectures like feedforward neural networks with non-polynomial activation functions, deep narrow networks with ReLU activation functions and functional input neural networks.
  arXiv  Detail & Related papers  (2025-10-10T09:21:34Z)
• Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization [66.03821840425539]
  In this paper, we investigate the training dynamics of $L$-layer neural networks using the tensor gradient program (SGD) framework.<n>We show that SGD enables these networks to learn linearly independent features that substantially deviate from their initial values.<n>This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum.
  arXiv  Detail & Related papers  (2025-03-12T17:33:13Z)
• Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study [8.183509993010983]
  We study the neural scaling laws for deep operator networks using the Chen and Chen style architecture. We quantify the neural scaling laws by analyzing its approximation and generalization errors. Our results offer a partial explanation of the neural scaling laws in operator learning and provide a theoretical foundation for their applications.
  arXiv  Detail & Related papers  (2024-10-01T03:06:55Z)
• Neural reproducing kernel Banach spaces and representer theorems for deep networks [14.902126718612648]
  We show that deep neural networks define reproducing suitable kernel Banach spaces.<n>We derive representer theorems that justify the finite architectures commonly employed in applications.
  arXiv  Detail & Related papers  (2024-03-13T17:51:02Z)
• Data Topology-Dependent Upper Bounds of Neural Network Widths [52.58441144171022]
  We first show that a three-layer neural network can be designed to approximate an indicator function over a compact set. This is then extended to a simplicial complex, deriving width upper bounds based on its topological structure. We prove the universal approximation property of three-layer ReLU networks using our topological approach.
  arXiv  Detail & Related papers  (2023-05-25T14:17:15Z)
• Approximation Power of Deep Neural Networks: an explanatory mathematical survey [0.0]
  The survey examines how effectively neural networks approximate target functions and to identify conditions under which they outperform traditional approximation methods.<n>Key topics include the nonlinear, compositional structure of deep networks and the formalization of neural network tasks as optimization problems in regression and classification settings.<n>The survey explores the density of neural networks in the space of continuous functions, comparing the approximation capabilities of deep ReLU networks with those of other approximation methods.
  arXiv  Detail & Related papers  (2022-07-19T18:47:44Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)
• Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics [50.83356836818667]
  We introduce a new theoretical framework to analyze deep learning optimization with connection to its generalization error. Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence.
  arXiv  Detail & Related papers  (2020-07-11T18:19:50Z)
• Expressivity of Deep Neural Networks [2.7909470193274593]
  In this review paper, we give a comprehensive overview of the large variety of approximation results for neural networks. While the mainbody of existing results is for general feedforward architectures, we also depict approximation results for convolutional, residual and recurrent neural networks.
  arXiv  Detail & Related papers  (2020-07-09T13:08:01Z)
• Understanding Generalization in Deep Learning via Tensor Methods [53.808840694241]
  We advance the understanding of the relations between the network's architecture and its generalizability from the compression perspective. We propose a series of intuitive, data-dependent and easily-measurable properties that tightly characterize the compressibility and generalizability of neural networks.
  arXiv  Detail & Related papers  (2020-01-14T22:26:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.