Representation formulas and pointwise properties for Barron functions

• URL: http://arxiv.org/abs/2006.05982v2
• Date: Fri, 4 Jun 2021 17:18:07 GMT
• Title: Representation formulas and pointwise properties for Barron functions
• Authors: Weinan E and Stephan Wojtowytsch
• Abstract summary: We study the natural function space for infinitely wide two-layer neural networks with ReLU activation (Barron space) We show that functions whose singular set is fractal or curved cannot be represented by infinitely wide two-layer networks with finite path-norm. This result suggests that two-layer neural networks may be able to approximate a greater variety of functions than commonly believed.
• Score: 8.160343645537106
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the natural function space for infinitely wide two-layer neural networks with ReLU activation (Barron space) and establish different representation formulae. In two cases, we describe the space explicitly up to isomorphism. Using a convenient representation, we study the pointwise properties of two-layer networks and show that functions whose singular set is fractal or curved (for example distance functions from smooth submanifolds) cannot be represented by infinitely wide two-layer networks with finite path-norm. We use this structure theorem to show that the only $C^1$-diffeomorphisms which Barron space are affine. Furthermore, we show that every Barron function can be decomposed as the sum of a bounded and a positively one-homogeneous function and that there exist Barron functions which decay rapidly at infinity and are globally Lebesgue-integrable. This result suggests that two-layer neural networks may be able to approximate a greater variety of functions than commonly believed.

Related papers

• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Functional Diffusion [55.251174506648454]
  We propose a new class of generative diffusion models, called functional diffusion. functional diffusion can be seen as an extension of classical diffusion models to an infinite-dimensional domain. We show generative results on complicated signed distance functions and deformation functions defined on 3D surfaces.
  arXiv  Detail & Related papers  (2023-11-26T21:35:34Z)
• Structure of universal formulas [13.794391803767617]
  We introduce a hierarchy of classes connecting the global approximability property to the weaker property of infinite VC dimension. We show that fixed-size neural networks with not more than one layer of neurons having activations cannot approximate functions on arbitrary finite sets. We give examples of functional families, including two-hidden-layer neural networks, that approximate functions on arbitrary finite sets, but fail to do that on the whole domain of definition.
  arXiv  Detail & Related papers  (2023-11-07T11:50:25Z)
• Barron Space for Graph Convolution Neural Networks [4.980329703241598]
  Graph convolutional neural network (GCNN) operates on graph domain. In this paper, we introduce a Barron space of functions on a compact domain of graph signals.
  arXiv  Detail & Related papers  (2023-11-06T02:58:05Z)
• Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions [63.21838830509772]
  We develop a framework for extending set functions onto low-dimensional continuous domains. Our framework subsumes many well-known extensions as special cases. We convert low-dimensional neural network bottlenecks into representations in high-dimensional spaces.
  arXiv  Detail & Related papers  (2022-08-08T10:58:02Z)
• Depth separation beyond radial functions [34.11708874794196]
  We show that certain functions can be efficiently approximated by two-hidden-layer networks but not by one-hidden-layer ones in high-dimensions $d$. A common theme in the proof of such results is the fact that one-hidden-layer networks fail to approximate high-energy functions whose Fourier representation is spread in the domain.
  arXiv  Detail & Related papers  (2021-02-02T17:25:02Z)
• Size and Depth Separation in Approximating Natural Functions with Neural Networks [52.73592689730044]
  We show the benefits of size and depth for approximation of natural functions with ReLU networks. We show a complexity-theoretic barrier to proving such results beyond size $O(d)$. We also show an explicit natural function, that can be approximated with networks of size $O(d)$.
  arXiv  Detail & Related papers  (2021-01-30T21:30:11Z)
• The Kolmogorov-Arnold representation theorem revisited [0.0]
  There is a longstanding debate whether the Kolmogorov-Arnold representation theorem can explain the use of more than one hidden layer in neural networks. We derive modifications of the Kolmogorov-Arnold representation that transfer smoothness properties of the represented function to the outer function and can be well approximated by ReLU networks. It appears that instead of two hidden layers, a more natural interpretation of the Kolmogorov-Arnold representation is that of a deep neural network where most of the layers are required to approximate the interior function.
  arXiv  Detail & Related papers  (2020-07-31T07:41:09Z)
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators [72.62940905965267]
  Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. Are CF-INNs universal approximators for invertible functions? We prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes.
  arXiv  Detail & Related papers  (2020-06-20T02:07:37Z)
• PDE constraints on smooth hierarchical functions computed by neural networks [0.0]
  An important problem in the theory of deep neural networks is expressivity. We study real infinitely differentiable (smooth) hierarchical functions implemented by feedforward neural networks. We conjecture that such PDE constraints, once accompanied by appropriate non-singularity conditions, guarantee that the smooth function under consideration can be represented by the network.
  arXiv  Detail & Related papers  (2020-05-18T16:34:11Z)
• Space of Functions Computed by Deep-Layered Machines [74.13735716675987]
  We study the space of functions computed by random-layered machines, including deep neural networks and Boolean circuits. Investigating the distribution of Boolean functions computed on the recurrent and layer-dependent architectures, we find that it is the same in both models.
  arXiv  Detail & Related papers  (2020-04-19T18:31:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.