Neural Tangent Kernel: A Survey

• URL: http://arxiv.org/abs/2208.13614v1
• Date: Mon, 29 Aug 2022 14:05:54 GMT
• Title: Neural Tangent Kernel: A Survey
• Authors: Eugene Golikov, Eduard Pokonechnyy, Vladimir Korviakov
• Abstract summary: A seminal work demonstrated that training a neural network under specific parameterization is equivalent to performing a particular kernel method as width goes to infinity. This equivalence opened a promising direction for applying the results of the rich literature on kernel methods to neural nets which were much harder to tackle. The present survey covers key results on kernel convergence as width goes to infinity, finite-width corrections, applications, and a discussion of the limitations of the corresponding method.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A seminal work [Jacot et al., 2018] demonstrated that training a neural network under specific parameterization is equivalent to performing a particular kernel method as width goes to infinity. This equivalence opened a promising direction for applying the results of the rich literature on kernel methods to neural nets which were much harder to tackle. The present survey covers key results on kernel convergence as width goes to infinity, finite-width corrections, applications, and a discussion of the limitations of the corresponding method.

Related papers

• Understanding the role of depth in the neural tangent kernel for overparameterized neural networks [14.953877391067893]
  We study the sensitivity of large Re parameterLU networks to increasing depths by characterizing the corresponding limiting kernel.<n>We empirically evaluate the order of magnitude in network depth required to observe this convergent behavior.
  arXiv  Detail & Related papers  (2025-11-10T16:18:04Z)
• Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernels [13.302913618949468]
  We provide a principled framework to interpret over- parameterized neural networks by mapping hierarchical feature transformations into kernel spaces.<n>We propose a depth-induced NTK kernel based on a shortcut-related architecture, which converges to a Gaussian process as the network depth approaches infinity.<n>Our findings significantly extend the existing landscape of the neural kernel theory and provide an in-depth understanding of deep learning and the scaling law.
  arXiv  Detail & Related papers  (2025-11-05T10:00:03Z)
• Random feature approximation for general spectral methods [2.9388890036358104]
  This work extends previous results for Tikhonov regularization to a broad class of spectral regularization techniques.<n>We enable a theoretical analysis of neural networks and neural operators through the lens of the Neural Tangent Kernel (NTK) approach.
  arXiv  Detail & Related papers  (2025-06-19T13:00:17Z)
• Adaptive kernel predictors from feature-learning infinite limits of neural networks [35.95321041944522]
  We show that neural networks trained in the rich, feature learning infinite-width regime are also described by kernel machines. We provide explicit expressions for the kernel predictors and prescriptions to numerically calculate them.
  arXiv  Detail & Related papers  (2025-02-11T22:34:49Z)
• Deep Kernel Posterior Learning under Infinite Variance Prior Weights [1.5960546024967326]
  We show that a Bayesian deep neural network converges to a process with $alpha$-stable marginals in each layer that has a conditionally Gaussian representation. We also provide useful generalizations of the results of Lor'ia & Bhadra (2024) on shallow networks on shallow multi-layer networks. The computational and statistical benefits over competing approaches stand out in simulations and in demonstrations on benchmark data sets.
  arXiv  Detail & Related papers  (2024-10-02T07:13:17Z)
• Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum [18.10812063219831]
  We introduce Modified Spectrum Kernels (MSKs) to approximate kernels with desired eigenvalues. We propose a preconditioned gradient descent method, which alters the trajectory of gradient descent. Our method is both computationally efficient and simple to implement.
  arXiv  Detail & Related papers  (2023-07-26T22:39:47Z)
• On the Sublinear Regret of GP-UCB [58.25014663727544]
  We show that the Gaussian Process Upper Confidence Bound (GP-UCB) algorithm enjoys nearly optimal regret rates. Our improvements rely on a key technical contribution -- regularizing kernel ridge estimators in proportion to the smoothness of the underlying kernel.
  arXiv  Detail & Related papers  (2023-07-14T13:56:11Z)
• A Simple Algorithm For Scaling Up Kernel Methods [0.0]
  We introduce a novel random feature regression algorithm that allows us to scale to virtually infinite numbers of random features. We illustrate the performance of our method on the CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2023-01-26T20:59:28Z)
• Zonotope Domains for Lagrangian Neural Network Verification [102.13346781220383]
  We decompose the problem of verifying a deep neural network into the verification of many 2-layer neural networks. Our technique yields bounds that improve upon both linear programming and Lagrangian-based verification techniques.
  arXiv  Detail & Related papers  (2022-10-14T19:31:39Z)
• An Overview of Uncertainty Quantification Methods for Infinite Neural Networks [0.0]
  We review methods for quantifying uncertainty in infinite-width neural networks. We make use of several equivalence results along the way to obtain exact closed-form solutions for predictive uncertainty.
  arXiv  Detail & Related papers  (2022-01-13T00:03:22Z)
• Random Features for the Neural Tangent Kernel [57.132634274795066]
  We propose an efficient feature map construction of the Neural Tangent Kernel (NTK) of fully-connected ReLU network. We show that dimension of the resulting features is much smaller than other baseline feature map constructions to achieve comparable error bounds both in theory and practice.
  arXiv  Detail & Related papers  (2021-04-03T09:08:12Z)
• Finite Versus Infinite Neural Networks: an Empirical Study [69.07049353209463]
  kernel methods outperform fully-connected finite-width networks. Centered and ensembled finite networks have reduced posterior variance. Weight decay and the use of a large learning rate break the correspondence between finite and infinite networks.
  arXiv  Detail & Related papers  (2020-07-31T01:57:47Z)
• Neural Splines: Fitting 3D Surfaces with Infinitely-Wide Neural Networks [61.07202852469595]
  We present Neural Splines, a technique for 3D surface reconstruction that is based on random feature kernels arising from infinitely-wide shallow ReLU networks. Our method achieves state-of-the-art results, outperforming recent neural network-based techniques and widely used Poisson Surface Reconstruction.
  arXiv  Detail & Related papers  (2020-06-24T14:54:59Z)
• On the infinite width limit of neural networks with a standard parameterization [52.07828272324366]
  We propose an improved extrapolation of the standard parameterization that preserves all of these properties as width is taken to infinity. We show experimentally that the resulting kernels typically achieve similar accuracy to those resulting from an NTK parameterization.
  arXiv  Detail & Related papers  (2020-01-21T01:02:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.