The Neural Tangent Kernel for Classification

• URL: http://arxiv.org/abs/2605.17606v2
• Date: Fri, 22 May 2026 23:38:49 GMT
• Title: The Neural Tangent Kernel for Classification
• Abstract summary: In wide neural networks, the Neural Tangent Kernel (NTK) remains approximately constant during training.<n>We show that parameter-space regularization ensures a constant NTK during training for cross-entropy loss.<n>In the absence of regularization the regime is recovered when targets are non-degenerate, i.e. when all classes have strictly positive probability.
• Score: 33.20539886208262
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In wide neural networks, the Neural Tangent Kernel (NTK) remains approximately constant during training, providing a powerful theoretical tool for studying training dynamics, generalization, and connections to kernel methods. However, this theory is largely restricted to regression losses. It was previously thought that training on a classification loss, or more generally losses involving nonlinear output transformations, breaks this property, leading to divergent logits and a breakdown of the linearization. In this paper, we extend NTK theory to classification by identifying conditions under which wide neural networks remain in the lazy training regime. We show that parameter-space regularization ensures a constant NTK during training for cross-entropy loss, while in the absence of regularization the regime is recovered when targets are non-degenerate, i.e. when all classes have strictly positive probability. Under these conditions, training is well-approximated by the linearized model, yielding an explicit characterization of the solution in terms of the NTK. We further analyze the distribution of trained predictors induced by random initialization and relate this notion of model uncertainty to Bayesian methods.

Related papers

• Divergence of Empirical Neural Tangent Kernel in Classification Problems [0.0]
  In classification problems, fully connected neural networks (FCNs) and residual neural networks (ResNets) cannot be approximated by kernel logistic regression based on the Neural Tangent Kernel (NTK)<n>We show that the empirical NTK does not uniformly converge to the NTK across all times on the training samples as the network width increases.
  arXiv  Detail & Related papers  (2025-04-15T12:30:21Z)
• Issues with Neural Tangent Kernel Approach to Neural Networks [13.710104651002869]
  We revisit the derivation of the NTK and conduct numerical experiments to evaluate this equivalence theorem.<n>We observe that adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in the predictor error.<n>These observations suggest the equivalence theorem does not hold well in practice and puts into question whether neural tangent kernels adequately address the training process of neural networks.
  arXiv  Detail & Related papers  (2025-01-19T03:21:06Z)
• The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks [0.0]
  We show how the NTK perspective falls short in the nonlinear scenario. We explore the convergence guarantees of such methods in both linear and nonlinear cases.
  arXiv  Detail & Related papers  (2024-02-06T10:24:36Z)
• Implicit regularization of deep residual networks towards neural ODEs [8.075122862553359]
  We establish an implicit regularization of deep residual networks towards neural ODEs. We prove that if the network is as a discretization of a neural ODE, then such a discretization holds throughout training.
  arXiv  Detail & Related papers  (2023-09-03T16:35:59Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets [57.06026574261203]
  We provide theoretical evidence for learning low-dimensional state spaces, which can also model long-term memory. Experiments corroborate our theory, demonstrating extrapolation via learning low-dimensional state spaces with both linear and non-linear RNNs.
  arXiv  Detail & Related papers  (2022-10-25T14:45:15Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)
• Frequentist Uncertainty in Recurrent Neural Networks via Blockwise Influence Functions [121.10450359856242]
  Recurrent neural networks (RNNs) are instrumental in modelling sequential and time-series data. Existing approaches for uncertainty quantification in RNNs are based predominantly on Bayesian methods. We develop a frequentist alternative that: (a) does not interfere with model training or compromise its accuracy, (b) applies to any RNN architecture, and (c) provides theoretical coverage guarantees on the estimated uncertainty intervals.
  arXiv  Detail & Related papers  (2020-06-20T22:45:32Z)
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks [87.23360438947114]
  We show that noisy gradient descent with weight decay can still exhibit a " Kernel-like" behavior. This implies that the training loss converges linearly up to a certain accuracy. We also establish a novel generalization error bound for two-layer neural networks trained by noisy gradient descent with weight decay.
  arXiv  Detail & Related papers  (2020-02-10T18:56:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.