A Gap Between the Gaussian RKHS and Neural Networks: An Infinite-Center Asymptotic Analysis

• URL: http://arxiv.org/abs/2502.16331v1
• Date: Sat, 22 Feb 2025 19:33:19 GMT
• Title: A Gap Between the Gaussian RKHS and Neural Networks: An Infinite-Center Asymptotic Analysis
• Authors: Akash Kumar, Rahul Parhi, Mikhail Belkin,
• Abstract summary: We show that certain functions that lie in the Gaussian RKHS have infinite norm in the neural network Banach space.<n>This provides a nontrivial gap between kernel methods and neural networks.
• Score: 18.454085925930073
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent works have characterized the function-space inductive bias of infinite-width bounded-norm single-hidden-layer neural networks as a kind of bounded-variation-type space. This novel neural network Banach space encompasses many classical multivariate function spaces including certain Sobolev spaces and the spectral Barron spaces. Notably, this Banach space also includes functions that exhibit less classical regularity such as those that only vary in a few directions. On bounded domains, it is well-established that the Gaussian reproducing kernel Hilbert space (RKHS) strictly embeds into this Banach space, demonstrating a clear gap between the Gaussian RKHS and the neural network Banach space. It turns out that when investigating these spaces on unbounded domains, e.g., all of $\mathbb{R}^d$, the story is fundamentally different. We establish the following fundamental result: Certain functions that lie in the Gaussian RKHS have infinite norm in the neural network Banach space. This provides a nontrivial gap between kernel methods and neural networks by the exhibition of functions in which kernel methods can do strictly better than neural networks.

Related papers

• Neural reproducing kernel Banach spaces and representer theorems for deep networks [16.279502878600184]
  We show that deep neural networks define suitable reproducing kernel Banach spaces. We derive representer theorems that justify the finite architectures commonly employed in applications.
  arXiv  Detail & Related papers  (2024-03-13T17:51:02Z)
• Space-Time Approximation with Shallow Neural Networks in Fourier Lebesgue spaces [1.74048653626208]
  We study the inclusion of anisotropic weighted Fourier-Lebesgue spaces in the Bochner-Sobolev spaces. We establish a bound on the approximation rate for functions from the anisotropic weighted Fourier-Lebesgue spaces and approximation via SNNs in the Bochner-Sobolev norm.
  arXiv  Detail & Related papers  (2023-12-13T19:02:27Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Tangent Bundle Filters and Neural Networks: from Manifolds to Cellular Sheaves and Back [114.01902073621577]
  We use the convolution to define tangent bundle filters and tangent bundle neural networks (TNNs) We discretize TNNs both in space and time domains, showing that their discrete counterpart is a principled variant of the recently introduced Sheaf Neural Networks. We numerically evaluate the effectiveness of the proposed architecture on a denoising task of a tangent vector field over the unit 2-sphere.
  arXiv  Detail & Related papers  (2022-10-26T21:55:45Z)
• Continuous percolation in a Hilbert space for a large system of qubits [58.720142291102135]
  The percolation transition is defined through the appearance of the infinite cluster. We show that the exponentially increasing dimensionality of the Hilbert space makes its covering by finite-size hyperspheres inefficient. Our approach to the percolation transition in compact metric spaces may prove useful for its rigorous treatment in other contexts.
  arXiv  Detail & Related papers  (2022-10-15T13:53:21Z)
• Algebraic function based Banach space valued ordinary and fractional neural network approximations [0.0]
  approximations are pointwise and of the uniform norm. The related Banach space valued feed-forward neural networks are with one hidden layer.
  arXiv  Detail & Related papers  (2022-02-11T20:08:52Z)
• Neural Operator: Learning Maps Between Function Spaces [75.93843876663128]
  We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We prove a universal approximation theorem for our proposed neural operator, showing that it can approximate any given nonlinear continuous operator. An important application for neural operators is learning surrogate maps for the solution operators of partial differential equations.
  arXiv  Detail & Related papers  (2021-08-19T03:56:49Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• Banach Space Representer Theorems for Neural Networks and Ridge Splines [17.12783792226575]
  We develop a variational framework to understand the properties of the functions learned by neural networks fit to data. We derive a representer theorem showing that finite-width, single-hidden layer neural networks are solutions to inverse problems.
  arXiv  Detail & Related papers  (2020-06-10T02:57:37Z)
• Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels [8.160343645537106]
  We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space. We show that reproducing kernel Hilbert spaces are poor $L2$-approximators for the class of two-layer neural networks in high dimension.
  arXiv  Detail & Related papers  (2020-05-21T17:40:38Z)
• Neural Operator: Graph Kernel Network for Partial Differential Equations [57.90284928158383]
  This work is to generalize neural networks so that they can learn mappings between infinite-dimensional spaces (operators) We formulate approximation of the infinite-dimensional mapping by composing nonlinear activation functions and a class of integral operators. Experiments confirm that the proposed graph kernel network does have the desired properties and show competitive performance compared to the state of the art solvers.
  arXiv  Detail & Related papers  (2020-03-07T01:56:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.