Universal approximation results for neural networks with non-polynomial activation function over non-compact domains

• URL: http://arxiv.org/abs/2410.14759v2
• Date: Wed, 23 Oct 2024 06:51:05 GMT
• Title: Universal approximation results for neural networks with non-polynomial activation function over non-compact domains
• Authors: Ariel Neufeld, Philipp Schmocker,
• Abstract summary: We derive universal approximation results for neural networks within function spaces over non-compact subsets of a Euclidean space. We provide some dimension-independent rates for approximating a function with sufficiently regular and integrable Fourier transform by neural networks with non-polynomial activation function.
• Score: 3.3379026542599934
• License:
• Abstract: In this paper, we generalize the universal approximation property of single-hidden-layer feed-forward neural networks beyond the classical formulation over compact domains. More precisely, by assuming that the activation function is non-polynomial, we derive universal approximation results for neural networks within function spaces over non-compact subsets of a Euclidean space, e.g., weighted spaces, $L^p$-spaces, and (weighted) Sobolev spaces over unbounded domains, where the latter includes the approximation of the (weak) derivatives. Furthermore, we provide some dimension-independent rates for approximating a function with sufficiently regular and integrable Fourier transform by neural networks with non-polynomial activation function.

Related papers

• Approximation and interpolation of deep neural networks [0.0]
  In the overparametrized regime, deep neural network provide universal approximations and can interpolate any data set. In the last section, we provide a practical probabilistic method of finding such a point under general conditions on the activation function.
  arXiv  Detail & Related papers  (2023-04-20T08:45:16Z)
• Deep neural network approximation of composite functions without the curse of dimensionality [0.0]
  In this article we identify a class of high-dimensional continuous functions that can be approximated by deep neural networks (DNNs) The functions in our class can be expressed as a potentially unbounded special functions which include products, maxima, and certain parallelized Lipschitz continuous functions.
  arXiv  Detail & Related papers  (2023-04-12T12:08:59Z)
• Approximation of Nonlinear Functionals Using Deep ReLU Networks [7.876115370275732]
  We investigate the approximation power of functional deep neural networks associated with the rectified linear unit (ReLU) activation function. In addition, we establish rates of approximation of the proposed functional deep ReLU networks under mild regularity conditions.
  arXiv  Detail & Related papers  (2023-04-10T08:10:11Z)
• Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions [63.21838830509772]
  We develop a framework for extending set functions onto low-dimensional continuous domains. Our framework subsumes many well-known extensions as special cases. We convert low-dimensional neural network bottlenecks into representations in high-dimensional spaces.
  arXiv  Detail & Related papers  (2022-08-08T10:58:02Z)
• Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations [2.15145758970292]
  We derive the required depth, width, and sparsity of a deep neural network to approximate any H"older smooth function up to a given approximation error in H"older norms. The latter feature is essential to control generalization errors in many statistical and machine learning applications.
  arXiv  Detail & Related papers  (2022-06-20T01:18:29Z)
• Exploring Linear Feature Disentanglement For Neural Networks [63.20827189693117]
  Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have achieved great success in neural networks (NNs) Due to the complex non-linear characteristic of samples, the objective of those activation functions is to project samples from their original feature space to a linear separable feature space. This phenomenon ignites our interest in exploring whether all features need to be transformed by all non-linear functions in current typical NNs.
  arXiv  Detail & Related papers  (2022-03-22T13:09:17Z)
• Deep neural network approximation of analytic functions [91.3755431537592]
  entropy bound for the spaces of neural networks with piecewise linear activation functions. We derive an oracle inequality for the expected error of the considered penalized deep neural network estimators.
  arXiv  Detail & Related papers  (2021-04-05T18:02:04Z)
• Quantitative Rates and Fundamental Obstructions to Non-Euclidean Universal Approximation with Deep Narrow Feed-Forward Networks [3.8073142980733]
  We quantify the number of narrow layers required for "deep geometric feed-forward neural networks" We find that both the global and local universal approximation guarantees can only coincide when approximating null-homotopic functions.
  arXiv  Detail & Related papers  (2021-01-13T23:29:40Z)
• Interval Universal Approximation for Neural Networks [47.767793120249095]
  We introduce the interval universal approximation (IUA) theorem. IUA shows that neural networks can approximate any continuous function $f$ as we have known for decades. We study the computational complexity of constructing neural networks that are amenable to precise interval analysis.
  arXiv  Detail & Related papers  (2020-07-12T20:43:56Z)
• Minimum Width for Universal Approximation [91.02689252671291]
  We prove that the minimum width required for the universal approximation of the $Lp$ functions is exactly $maxd_x+1,d_y$. We also prove that the same conclusion does not hold for the uniform approximation with ReLU, but does hold with an additional threshold activation function.
  arXiv  Detail & Related papers  (2020-06-16T01:24:21Z)
• Neural Operator: Graph Kernel Network for Partial Differential Equations [57.90284928158383]
  This work is to generalize neural networks so that they can learn mappings between infinite-dimensional spaces (operators) We formulate approximation of the infinite-dimensional mapping by composing nonlinear activation functions and a class of integral operators. Experiments confirm that the proposed graph kernel network does have the desired properties and show competitive performance compared to the state of the art solvers.
  arXiv  Detail & Related papers  (2020-03-07T01:56:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.