Approximation by non-symmetric networks for cross-domain learning

• URL: http://arxiv.org/abs/2305.03890v2
• Date: Fri, 5 Jan 2024 05:14:13 GMT
• Title: Approximation by non-symmetric networks for cross-domain learning
• Authors: Hrushikesh Mhaskar
• Abstract summary: We study the approximation capabilities of kernel based networks using non-symmetric kernels. We obtain estimates on the accuracy of uniform approximation of functions in a ($L2$)-Sobolev class by ReLU$r$ networks when $r$ is not necessarily an integer.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For the past 30 years or so, machine learning has stimulated a great deal of research in the study of approximation capabilities (expressive power) of a multitude of processes, such as approximation by shallow or deep neural networks, radial basis function networks, and a variety of kernel based methods. Motivated by applications such as invariant learning, transfer learning, and synthetic aperture radar imaging, we initiate in this paper a general approach to study the approximation capabilities of kernel based networks using non-symmetric kernels. While singular value decomposition is a natural instinct to study such kernels, we consider a more general approach to include the use of a family of kernels, such as generalized translation networks (which include neural networks and translation invariant kernels as special cases) and rotated zonal function kernels. Naturally, unlike traditional kernel based approximation, we cannot require the kernels to be positive definite. In particular, we obtain estimates on the accuracy of uniform approximation of functions in a ($L^2$)-Sobolev class by ReLU$^r$ networks when $r$ is not necessarily an integer. Our general results apply to the approximation of functions with small smoothness compared to the dimension of the input space.

Related papers

• An Exact Kernel Equivalence for Finite Classification Models [1.4777718769290527]
  We compare our exact representation to the well-known Neural Tangent Kernel (NTK) and discuss approximation error relative to the NTK. We use this exact kernel to show that our theoretical contribution can provide useful insights into the predictions made by neural networks.
  arXiv  Detail & Related papers  (2023-08-01T20:22:53Z)
• On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains [10.360517127652185]
  We provide a strategy to determine the eigenvalue decay rate (EDR) of a large class of kernel functions defined on a general domain. This class of kernel functions include but are not limited to the neural tangent kernel associated with neural networks with different depths and various activation functions.
  arXiv  Detail & Related papers  (2023-05-04T08:54:40Z)
• Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK [86.45209429863858]
  We study training one-hidden-layer ReLU networks in the neural tangent kernel (NTK) regime. We show that the neural networks possess a different limiting kernel which we call textitbias-generalized NTK We also study various properties of the neural networks with this new kernel.
  arXiv  Detail & Related papers  (2023-01-01T02:11:39Z)
• Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth [92.25666446274188]
  sinusoidal neural networks with activations have been proposed as an alternative to networks with traditional activation functions. We first propose a simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis. We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kernel approximates a low-pass filter with an adjustable bandwidth.
  arXiv  Detail & Related papers  (2022-11-26T07:41:48Z)
• NeuralEF: Deconstructing Kernels by Deep Neural Networks [47.54733625351363]
  Traditional nonparametric solutions based on the Nystr"om formula suffer from scalability issues. Recent work has resorted to a parametric approach, i.e., training neural networks to approximate the eigenfunctions. We show that these problems can be fixed by using a new series of objective functions that generalizes to space of supervised and unsupervised learning problems.
  arXiv  Detail & Related papers  (2022-04-30T05:31:07Z)
• Inducing Gaussian Process Networks [80.40892394020797]
  We propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points. The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains. We report on experimental results for real-world data sets showing that IGNs provide significant advances over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-04-21T05:27:09Z)
• Universality and Optimality of Structured Deep Kernel Networks [0.0]
  Kernel based methods yield approximation models that are flexible, efficient and powerful. Recent success of machine learning methods has been driven by deep neural networks (NNs) In this paper, we show that the use of special types of kernels yield models reminiscent of neural networks.
  arXiv  Detail & Related papers  (2021-05-15T14:10:35Z)
• Random Features for the Neural Tangent Kernel [57.132634274795066]
  We propose an efficient feature map construction of the Neural Tangent Kernel (NTK) of fully-connected ReLU network. We show that dimension of the resulting features is much smaller than other baseline feature map constructions to achieve comparable error bounds both in theory and practice.
  arXiv  Detail & Related papers  (2021-04-03T09:08:12Z)
• Deep Equals Shallow for ReLU Networks in Kernel Regimes [13.909388235627791]
  We show that for ReLU activations, the kernels derived from deep fully-connected networks have essentially the same approximation properties as their shallow two-layer counterpart. Our main theoretical result relies on characterizing such eigenvalue decays through differentiability properties of the kernel function.
  arXiv  Detail & Related papers  (2020-09-30T02:37:43Z)
• Finite Versus Infinite Neural Networks: an Empirical Study [69.07049353209463]
  kernel methods outperform fully-connected finite-width networks. Centered and ensembled finite networks have reduced posterior variance. Weight decay and the use of a large learning rate break the correspondence between finite and infinite networks.
  arXiv  Detail & Related papers  (2020-07-31T01:57:47Z)
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks [12.692279981822011]
  We derive the covariance functions of multi-layer perceptrons with exponential linear units (ELU) and Gaussian error linear units (GELU) We analyse the fixed-point dynamics of iterated kernels corresponding to a broad range of activation functions. We find that unlike some previously studied neural network kernels, these new kernels exhibit non-trivial fixed-point dynamics.
  arXiv  Detail & Related papers  (2020-02-20T01:25:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.