Structure of universal formulas

• URL: http://arxiv.org/abs/2311.03910v1
• Date: Tue, 7 Nov 2023 11:50:25 GMT
• Title: Structure of universal formulas
• Authors: Dmitry Yarotsky
• Abstract summary: We introduce a hierarchy of classes connecting the global approximability property to the weaker property of infinite VC dimension. We show that fixed-size neural networks with not more than one layer of neurons having activations cannot approximate functions on arbitrary finite sets. We give examples of functional families, including two-hidden-layer neural networks, that approximate functions on arbitrary finite sets, but fail to do that on the whole domain of definition.
• Score: 13.794391803767617
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: By universal formulas we understand parameterized analytic expressions that have a fixed complexity, but nevertheless can approximate any continuous function on a compact set. There exist various examples of such formulas, including some in the form of neural networks. In this paper we analyze the essential structural elements of these highly expressive models. We introduce a hierarchy of expressiveness classes connecting the global approximability property to the weaker property of infinite VC dimension, and prove a series of classification results for several increasingly complex functional families. In particular, we introduce a general family of polynomially-exponentially-algebraic functions that, as we prove, is subject to polynomial constraints. As a consequence, we show that fixed-size neural networks with not more than one layer of neurons having transcendental activations (e.g., sine or standard sigmoid) cannot in general approximate functions on arbitrary finite sets. On the other hand, we give examples of functional families, including two-hidden-layer neural networks, that approximate functions on arbitrary finite sets, but fail to do that on the whole domain of definition.

Related papers

• Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensions [2.7195102129095003]
  Deep neural networks employing ReLU and ReLU$2$ activation functions can effectively represent Lagrange finite element functions of any order on various simplicial meshes in arbitrary dimensions. Our findings present the first demonstration of how deep neural networks can systematically generate general continuous piecewise functions on both specific or arbitrary simplicial meshes.
  arXiv  Detail & Related papers  (2023-12-21T19:57:29Z)
• Bayes Complexity of Learners vs Overfitting [4.873362301533825]
  We show that a new notion of complexity of functions governs a PAC Bayes-like generalization bound. In contrast to previous works, our notion naturally generalizes to neural networks with several layers. An upper-bound we derive allows to show a separation in the number of samples needed for good generalization between 2 and 4-layer neural networks.
  arXiv  Detail & Related papers  (2023-03-13T13:07:02Z)
• Qualitative neural network approximation over R and C: Elementary proofs for analytic and polynomial activation [0.0]
  We prove approximations in classes of deep and shallow neural networks with analytic activation functions. We show that fully connected and residual networks of large depth with activation functions can approximate any under certain width requirements.
  arXiv  Detail & Related papers  (2022-03-25T01:36:13Z)
• Dist2Cycle: A Simplicial Neural Network for Homology Localization [66.15805004725809]
  Simplicial complexes can be viewed as high dimensional generalizations of graphs that explicitly encode multi-way ordered relations. We propose a graph convolutional model for learning functions parametrized by the $k$-homological features of simplicial complexes.
  arXiv  Detail & Related papers  (2021-10-28T14:59:41Z)
• Bilinear Classes: A Structural Framework for Provable Generalization in RL [119.42509700822484]
  Bilinear Classes is a new structural framework which permits generalization in reinforcement learning. The framework incorporates nearly all existing models in which a sample complexity is achievable. Our main result provides an RL algorithm which has sample complexity for Bilinear Classes.
  arXiv  Detail & Related papers  (2021-03-19T16:34:20Z)
• The universal approximation theorem for complex-valued neural networks [0.0]
  We generalize the classical universal approximation for neural networks to the case of complex-valued neural networks. We consider feedforward networks with a complex activation function $sigma : mathbbC to mathbbC$ in which each neuron performs the operation $mathbbCN to mathbbC, z mapsto sigma(b + wT z)$ with weights $w in mathbbCN$ and a bias $b in math
  arXiv  Detail & Related papers  (2020-12-06T18:51:10Z)
• Finite-Function-Encoding Quantum States [52.77024349608834]
  We introduce finite-function-encoding (FFE) states which encode arbitrary $d$-valued logic functions. We investigate some of their structural properties.
  arXiv  Detail & Related papers  (2020-12-01T13:53:23Z)
• UNIPoint: Universally Approximating Point Processes Intensities [125.08205865536577]
  We provide a proof that a class of learnable functions can universally approximate any valid intensity function. We implement UNIPoint, a novel neural point process model, using recurrent neural networks to parameterise sums of basis function upon each event.
  arXiv  Detail & Related papers  (2020-07-28T09:31:56Z)
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators [72.62940905965267]
  Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. Are CF-INNs universal approximators for invertible functions? We prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes.
  arXiv  Detail & Related papers  (2020-06-20T02:07:37Z)
• PDE constraints on smooth hierarchical functions computed by neural networks [0.0]
  An important problem in the theory of deep neural networks is expressivity. We study real infinitely differentiable (smooth) hierarchical functions implemented by feedforward neural networks. We conjecture that such PDE constraints, once accompanied by appropriate non-singularity conditions, guarantee that the smooth function under consideration can be represented by the network.
  arXiv  Detail & Related papers  (2020-05-18T16:34:11Z)
• Invariant Feature Coding using Tensor Product Representation [75.62232699377877]
  We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier. A novel feature model that explicitly consider group action is proposed for principal component analysis and k-means clustering.
  arXiv  Detail & Related papers  (2019-06-05T07:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.