Understanding neural networks with reproducing kernel Banach spaces

• URL: http://arxiv.org/abs/2109.09710v1
• Date: Mon, 20 Sep 2021 17:32:30 GMT
• Title: Understanding neural networks with reproducing kernel Banach spaces
• Authors: Francesca Bartolucci, Ernesto De Vito, Lorenzo Rosasco, Stefano Vigogna
• Abstract summary: Characterizing function spaces corresponding to neural networks can provide a way to understand their properties. We prove a representer theorem for a wide class of reproducing kernel Banach spaces. For a suitable class of ReLU activation functions, the norm in the corresponding kernel Banach space can be characterized in terms of the inverse Radon transform of a bounded real measure.
• Score: 20.28372804772848
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Characterizing the function spaces corresponding to neural networks can provide a way to understand their properties. In this paper we discuss how the theory of reproducing kernel Banach spaces can be used to tackle this challenge. In particular, we prove a representer theorem for a wide class of reproducing kernel Banach spaces that admit a suitable integral representation and include one hidden layer neural networks of possibly infinite width. Further, we show that, for a suitable class of ReLU activation functions, the norm in the corresponding reproducing kernel Banach space can be characterized in terms of the inverse Radon transform of a bounded real measure, with norm given by the total variation norm of the measure. Our analysis simplifies and extends recent results in [34,29,30].

Related papers

• A Non-negative VAE:the Generalized Gamma Belief Network [49.970917207211556]
  The gamma belief network (GBN) has demonstrated its potential for uncovering multi-layer interpretable latent representations in text data. We introduce the generalized gamma belief network (Generalized GBN) in this paper, which extends the original linear generative model to a more expressive non-linear generative model. We also propose an upward-downward Weibull inference network to approximate the posterior distribution of the latent variables.
  arXiv  Detail & Related papers  (2024-08-06T18:18:37Z)
• Novel Kernel Models and Exact Representor Theory for Neural Networks Beyond the Over-Parameterized Regime [52.00917519626559]
  This paper presents two models of neural-networks and their training applicable to neural networks of arbitrary width, depth and topology. We also present an exact novel representor theory for layer-wise neural network training with unregularized gradient descent in terms of a local-extrinsic neural kernel (LeNK) This representor theory gives insight into the role of higher-order statistics in neural network training and the effect of kernel evolution in neural-network kernel models.
  arXiv  Detail & Related papers  (2024-05-24T06:30:36Z)
• Neural reproducing kernel Banach spaces and representer theorems for deep networks [16.279502878600184]
  We show that deep neural networks define suitable reproducing kernel Banach spaces. We derive representer theorems that justify the finite architectures commonly employed in applications.
  arXiv  Detail & Related papers  (2024-03-13T17:51:02Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks [49.870593940818715]
  We study the infinite-width limit of a type of three-layer NN model whose first layer is random and fixed. Our theory accommodates different scaling choices of the model, resulting in two regimes of the MF limit that demonstrate distinctive behaviors.
  arXiv  Detail & Related papers  (2022-10-28T17:26:27Z)
• Unified Fourier-based Kernel and Nonlinearity Design for Equivariant Networks on Homogeneous Spaces [52.424621227687894]
  We introduce a unified framework for group equivariant networks on homogeneous spaces. We take advantage of the sparsity of Fourier coefficients of the lifted feature fields. We show that other methods treating features as the Fourier coefficients in the stabilizer subgroup are special cases of our activation.
  arXiv  Detail & Related papers  (2022-06-16T17:59:01Z)
• Ridges, Neural Networks, and the Radon Transform [25.6264886382888]
  Ridges appear in the theory of neural networks as functional descriptors of the effect of a neuron. We investigate properties of the Radon transform in relation to ridges and to the characterization of neural networks.
  arXiv  Detail & Related papers  (2022-03-04T19:38:15Z)
• Two-layer neural networks with values in a Banach space [1.90365714903665]
  We study two-layer neural networks whose domain and range are Banach spaces with separable preduals. As the nonlinearity we choose the lattice operation of taking the positive part; in case of $mathbb Rd$-valued neural networks this corresponds to the ReLU activation function.
  arXiv  Detail & Related papers  (2021-05-05T14:54:24Z)
• Approximations with deep neural networks in Sobolev time-space [5.863264019032882]
  Solution of evolution equation generally lies in certain Bochner-Sobolev spaces. Deep neural networks can approximate Sobolev-regular functions with respect to Bochner-Sobolev spaces.
  arXiv  Detail & Related papers  (2020-12-23T22:21:05Z)
• Neural Splines: Fitting 3D Surfaces with Infinitely-Wide Neural Networks [61.07202852469595]
  We present Neural Splines, a technique for 3D surface reconstruction that is based on random feature kernels arising from infinitely-wide shallow ReLU networks. Our method achieves state-of-the-art results, outperforming recent neural network-based techniques and widely used Poisson Surface Reconstruction.
  arXiv  Detail & Related papers  (2020-06-24T14:54:59Z)
• Banach Space Representer Theorems for Neural Networks and Ridge Splines [17.12783792226575]
  We develop a variational framework to understand the properties of the functions learned by neural networks fit to data. We derive a representer theorem showing that finite-width, single-hidden layer neural networks are solutions to inverse problems.
  arXiv  Detail & Related papers  (2020-06-10T02:57:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.