An Exact Kernel Equivalence for Finite Classification Models

• URL: http://arxiv.org/abs/2308.00824v3
• Date: Wed, 9 Aug 2023 16:25:24 GMT
• Title: An Exact Kernel Equivalence for Finite Classification Models
• Authors: Brian Bell, Michael Geyer, David Glickenstein, Amanda Fernandez, Juston Moore
• Abstract summary: We compare our exact representation to the well-known Neural Tangent Kernel (NTK) and discuss approximation error relative to the NTK. We use this exact kernel to show that our theoretical contribution can provide useful insights into the predictions made by neural networks.
• Score: 1.4777718769290527
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We explore the equivalence between neural networks and kernel methods by deriving the first exact representation of any finite-size parametric classification model trained with gradient descent as a kernel machine. We compare our exact representation to the well-known Neural Tangent Kernel (NTK) and discuss approximation error relative to the NTK and other non-exact path kernel formulations. We experimentally demonstrate that the kernel can be computed for realistic networks up to machine precision. We use this exact kernel to show that our theoretical contribution can provide useful insights into the predictions made by neural networks, particularly the way in which they generalize.

Related papers

• Novel Kernel Models and Exact Representor Theory for Neural Networks Beyond the Over-Parameterized Regime [52.00917519626559]
  This paper presents two models of neural-networks and their training applicable to neural networks of arbitrary width, depth and topology. We also present an exact novel representor theory for layer-wise neural network training with unregularized gradient descent in terms of a local-extrinsic neural kernel (LeNK) This representor theory gives insight into the role of higher-order statistics in neural network training and the effect of kernel evolution in neural-network kernel models.
  arXiv  Detail & Related papers  (2024-05-24T06:30:36Z)
• Neural Tangent Kernels Motivate Graph Neural Networks with Cross-Covariance Graphs [94.44374472696272]
  We investigate NTKs and alignment in the context of graph neural networks (GNNs) Our results establish the theoretical guarantees on the optimality of the alignment for a two-layer GNN. These guarantees are characterized by the graph shift operator being a function of the cross-covariance between the input and the output data.
  arXiv  Detail & Related papers  (2023-10-16T19:54:21Z)
• Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models [7.608408123113268]
  We analyze approximate empirical neural tangent kernels (eNTK) for data attribution. We introduce two new random projection variants of approximate eNTK which allow users to tune the time and memory complexity of their calculation. We conclude that kernel machines using approximate neural tangent kernel as the kernel function are effective surrogate models.
  arXiv  Detail & Related papers  (2023-05-23T23:51:53Z)
• Approximation by non-symmetric networks for cross-domain learning [0.0]
  We study the approximation capabilities of kernel based networks using non-symmetric kernels. We obtain estimates on the accuracy of uniform approximation of functions in a Sobolev class by ReLU$r$ networks when $r$ is not necessarily an integer.
  arXiv  Detail & Related papers  (2023-05-06T01:33:26Z)
• On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains [10.360517127652185]
  We provide a strategy to determine the eigenvalue decay rate (EDR) of a large class of kernel functions defined on a general domain. This class of kernel functions include but are not limited to the neural tangent kernel associated with neural networks with different depths and various activation functions.
  arXiv  Detail & Related papers  (2023-05-04T08:54:40Z)
• Uniform Generalization Bounds for Overparameterized Neural Networks [5.945320097465419]
  We prove uniform generalization bounds for overparameterized neural networks in kernel regimes. Our bounds capture the exact error rates depending on the differentiability of the activation functions. We show the equivalence between the RKHS corresponding to the NT kernel and its counterpart corresponding to the Mat'ern family of kernels.
  arXiv  Detail & Related papers  (2021-09-13T16:20:13Z)
• Random Features for the Neural Tangent Kernel [57.132634274795066]
  We propose an efficient feature map construction of the Neural Tangent Kernel (NTK) of fully-connected ReLU network. We show that dimension of the resulting features is much smaller than other baseline feature map constructions to achieve comparable error bounds both in theory and practice.
  arXiv  Detail & Related papers  (2021-04-03T09:08:12Z)
• Finite Versus Infinite Neural Networks: an Empirical Study [69.07049353209463]
  kernel methods outperform fully-connected finite-width networks. Centered and ensembled finite networks have reduced posterior variance. Weight decay and the use of a large learning rate break the correspondence between finite and infinite networks.
  arXiv  Detail & Related papers  (2020-07-31T01:57:47Z)
• Neural Splines: Fitting 3D Surfaces with Infinitely-Wide Neural Networks [61.07202852469595]
  We present Neural Splines, a technique for 3D surface reconstruction that is based on random feature kernels arising from infinitely-wide shallow ReLU networks. Our method achieves state-of-the-art results, outperforming recent neural network-based techniques and widely used Poisson Surface Reconstruction.
  arXiv  Detail & Related papers  (2020-06-24T14:54:59Z)
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks [87.23360438947114]
  We show that noisy gradient descent with weight decay can still exhibit a " Kernel-like" behavior. This implies that the training loss converges linearly up to a certain accuracy. We also establish a novel generalization error bound for two-layer neural networks trained by noisy gradient descent with weight decay.
  arXiv  Detail & Related papers  (2020-02-10T18:56:15Z)
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks [17.188280334580195]
  We derive analytical expressions for the generalization performance of kernel regression as a function of the number of training samples. Our expressions apply to wide neural networks due to an equivalence between training them and kernel regression with the Neural Kernel Tangent (NTK) We verify our theory with simulations on synthetic data and MNIST dataset.
  arXiv  Detail & Related papers  (2020-02-07T00:03:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.